May 26 15:58:41.284642 ip-10-0-137-193 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 26 15:58:41.284654 ip-10-0-137-193 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 26 15:58:41.284661 ip-10-0-137-193 systemd[1]: kubelet.service: Failed with result 'resources'.
May 26 15:58:41.284901 ip-10-0-137-193 systemd[1]: Failed to start Kubernetes Kubelet.
May 26 15:58:51.456732 ip-10-0-137-193 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 26 15:58:51.456748 ip-10-0-137-193 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ea355fededca4171abf978a64ef2faf3 --
May 26 16:01:22.463658 ip-10-0-137-193 systemd[1]: Starting Kubernetes Kubelet...
May 26 16:01:22.904404 ip-10-0-137-193 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:01:22.904404 ip-10-0-137-193 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 26 16:01:22.904404 ip-10-0-137-193 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:01:22.904404 ip-10-0-137-193 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 26 16:01:22.904404 ip-10-0-137-193 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:01:22.905475 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.905363    2560 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 26 16:01:22.908614 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908594    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:01:22.908614 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908610    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:01:22.908614 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908615    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:01:22.908614 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908619    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908623    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908628    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908631    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908635    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908640    2560 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908644    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908647    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908651    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908654    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908659    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908663    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908667    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908671    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908675    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908679    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908683    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908690    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908695    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:01:22.908862 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908699    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908704    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908709    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908713    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908717    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908725    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908729    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908733    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908738    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908744    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908748    2560 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908752    2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908757    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908761    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908765    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908770    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908774    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908778    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908782    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908787    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:01:22.909709 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908791    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908796    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908800    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908805    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908809    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908813    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908817    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908821    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908825    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908829    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908834    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908839    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908843    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908848    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908854    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908858    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908861    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908866    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908874    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908880    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:01:22.910594 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908884    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908888    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908893    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908897    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908901    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908905    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908909    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908913    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908918    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908922    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908926    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908930    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908935    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908948    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908952    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908957    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908961    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908965    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908968    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:01:22.911472 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908973    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908977    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908981    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908985    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.908990    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909658    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909669    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909674    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909679    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909684    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909688    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909692    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909696    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909700    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909704    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909708    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909712    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909716    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909720    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909724    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:01:22.911992 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909728    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909732    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909737    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909741    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909745    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909749    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909754    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909759    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909763    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909767    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909772    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909776    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909780    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909785    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909789    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909793    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909797    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909802    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909806    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909811    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:01:22.912578 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909815    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909819    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909823    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909827    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909832    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909836    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909841    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909845    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909849    2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909853    2560 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909858    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909862    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909867    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909871    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909875    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909879    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909884    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909888    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909892    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:01:22.913133 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909897    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909901    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909905    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909909    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909914    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909918    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909922    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909926    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909930    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909934    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909939    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909943    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909947    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909952    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909956    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909966    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909970    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909974    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909978    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909982    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:01:22.913829 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909988    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909994    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.909999    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.910003    2560 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.910007    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.910013    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.910017    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.910023    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.910030    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.910035    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.910039    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.910044    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910138    2560 flags.go:64] FLAG: --address="0.0.0.0"
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910150    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910160    2560 flags.go:64] FLAG: --anonymous-auth="true"
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910167    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910174    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910179    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910186    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910195    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910203    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 26 16:01:22.914316 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910212    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910222    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910229    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910235    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910240    2560 flags.go:64] FLAG: --cgroup-root=""
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910245    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910249    2560 flags.go:64] FLAG: --client-ca-file=""
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910254    2560 flags.go:64] FLAG: --cloud-config=""
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910259    2560 flags.go:64] FLAG: --cloud-provider="external"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910264    2560 flags.go:64] FLAG: --cluster-dns="[]"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910274    2560 flags.go:64] FLAG: --cluster-domain=""
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910279    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910284    2560 flags.go:64] FLAG: --config-dir=""
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910289    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910294    2560 flags.go:64] FLAG: --container-log-max-files="5"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910300    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910306    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910311    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910317    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910321    2560 flags.go:64] FLAG: --contention-profiling="false"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910326    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910331    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910336    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910341    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910347    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 26 16:01:22.914890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910353    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910358    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910371    2560 flags.go:64] FLAG: --enable-load-reader="false"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910377    2560 flags.go:64] FLAG: --enable-server="true"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910381    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910403    2560 flags.go:64] FLAG: --event-burst="100"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910409    2560 flags.go:64] FLAG: --event-qps="50"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910414    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910419    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910424    2560 flags.go:64] FLAG: --eviction-hard=""
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910430    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910435    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910441    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910445    2560 flags.go:64] FLAG: --eviction-soft=""
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910450    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910455    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910460    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910465    2560 flags.go:64] FLAG: --experimental-mounter-path=""
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910470    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910475    2560 flags.go:64] FLAG: --fail-swap-on="true"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910480    2560 flags.go:64] FLAG: --feature-gates=""
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910486    2560 flags.go:64] FLAG: --file-check-frequency="20s"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910491    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910496    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910502    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910507    2560 flags.go:64] FLAG: --healthz-port="10248"
May 26 16:01:22.915584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910512    2560 flags.go:64] FLAG: --help="false"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910517    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-137-193.ec2.internal"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910522    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910528    2560 flags.go:64] FLAG: --http-check-frequency="20s"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910532    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910538    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910543    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910550    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910555    2560 flags.go:64] FLAG: --image-service-endpoint=""
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910560    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910565    2560 flags.go:64] FLAG: --kube-api-burst="100"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910570    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910575    2560 flags.go:64] FLAG: --kube-api-qps="50"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910580    2560 flags.go:64] FLAG: --kube-reserved=""
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910585    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910590    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910595    2560 flags.go:64] FLAG: --kubelet-cgroups=""
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910599    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910603    2560 flags.go:64] FLAG: --lock-file=""
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910608    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910613    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910618    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910627    2560 flags.go:64] FLAG: --log-json-split-stream="false"
May 26 16:01:22.916955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910632    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910637    2560 flags.go:64] FLAG: --log-text-split-stream="false"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910642    2560 flags.go:64] FLAG: --logging-format="text"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910647    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910653    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910658    2560 flags.go:64] FLAG: --manifest-url=""
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910662    2560 flags.go:64] FLAG: --manifest-url-header=""
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910669    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910674    2560 flags.go:64] FLAG: --max-open-files="1000000"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910681    2560 flags.go:64] FLAG: --max-pods="110"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910686    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910691    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910696    2560 flags.go:64] FLAG: --memory-manager-policy="None"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910701    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910706    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910711    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910717    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910729    2560 flags.go:64] FLAG: --node-status-max-images="50"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910734    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910739    2560 flags.go:64] FLAG: --oom-score-adj="-999"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910744    2560 flags.go:64] FLAG: --pod-cidr=""
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910749    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910758    2560 flags.go:64] FLAG: --pod-manifest-path=""
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910763    2560 flags.go:64] FLAG: --pod-max-pids="-1"
May 26 16:01:22.917609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910768    2560 flags.go:64] FLAG: --pods-per-core="0"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910773    2560 flags.go:64] FLAG: --port="10250"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910778    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910783    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a012a80713cb3440"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910789    2560 flags.go:64] FLAG: --qos-reserved=""
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910794    2560 flags.go:64] FLAG: --read-only-port="10255"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910799    2560 flags.go:64] FLAG: --register-node="true"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910804    2560 flags.go:64] FLAG: --register-schedulable="true"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910809    2560 flags.go:64] FLAG: --register-with-taints=""
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910815    2560 flags.go:64] FLAG: --registry-burst="10"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910821    2560 flags.go:64] FLAG: --registry-qps="5"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910825    2560 flags.go:64] FLAG: --reserved-cpus=""
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910830    2560 flags.go:64] FLAG: --reserved-memory=""
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910841    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910846    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910851    2560 flags.go:64] FLAG: --rotate-certificates="false"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910860    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910865    2560 flags.go:64] FLAG: --runonce="false"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910869    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910874    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910879    2560 flags.go:64] FLAG: --seccomp-default="false"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910884    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910889    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910894    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910898    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910903    2560 flags.go:64] FLAG: --storage-driver-password="root"
May 26 16:01:22.918227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910909    2560 flags.go:64] FLAG: --storage-driver-secure="false"
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910913    2560 flags.go:64] FLAG: --storage-driver-table="stats"
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910918    2560 flags.go:64] FLAG: --storage-driver-user="root"
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910923    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910928    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910933    2560 flags.go:64] FLAG: --system-cgroups=""
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910938    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910947    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910952    2560 flags.go:64] FLAG: --tls-cert-file=""
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910956    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910964    2560 flags.go:64] FLAG: --tls-min-version=""
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910969    2560 flags.go:64] FLAG: --tls-private-key-file=""
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910973    2560 flags.go:64] FLAG: --topology-manager-policy="none"
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910978    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910983    2560 flags.go:64] FLAG: --topology-manager-scope="container"
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910991    2560 flags.go:64] FLAG: --v="2"
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.910997    2560 flags.go:64] FLAG: --version="false"
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.911004    2560 flags.go:64] FLAG: --vmodule=""
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.911011    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.911017    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911165    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911172    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911179    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911184    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:01:22.918888 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911188    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911192    2560 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911196    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911201    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911205    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911209    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911213    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911218    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911222    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911227    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911231    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911238    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911244    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911250    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911255    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911259    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911263    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911267    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911271    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:01:22.919478 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911275    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911279    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911284    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911288    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911294    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911298    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911302    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911307    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911311    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911315    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911319    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911325    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911329    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911333    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911338    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911342    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911346    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911350    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911354    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911358    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:01:22.919981 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911362    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911367    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911371    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911376    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911381    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911385    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911408    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911412    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911416    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911421    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911425    2560 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911429    2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911433    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911437    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911442    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911446    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911452    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911456    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911461    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911465    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:01:22.920506 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911470    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911474    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911478    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911483    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911488    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911491    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911496    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911500    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911505    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911509    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911515    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911521    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911527    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911531    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911535    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911539    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911549    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911554    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911559    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:01:22.920995 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911563    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:01:22.921485 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911567    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:01:22.921485 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911572    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:01:22.921485 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.911576    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:01:22.921485 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.912250    2560 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:01:22.921712 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.921694    2560 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 26 16:01:22.921740 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.921713    2560 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 26 16:01:22.921773 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921765    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:01:22.921773 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921773    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921777    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921781    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921784    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921786    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921789    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921792    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921794    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921797    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921800    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921803    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921806    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921808    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921811    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921813    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921816    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921819    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921821    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921824    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:01:22.921826 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921827    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921829    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921832    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921835    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921838    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921841    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921844    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921846    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921849    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921851    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921854    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921857    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921859    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921862    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921866    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921869    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921872    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921875    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921877    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921880    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:01:22.922301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921882    2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921885    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921887    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921889    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921892    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921894    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921897    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921899    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921902    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921904    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921907    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921909    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921912    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921915    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921918    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921920    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921923    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921925    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921928    2560 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921931    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:01:22.922808 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921933    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921936    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921938    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921941    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921944    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921946    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921949    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921951    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921956    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921960    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921963    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921966    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921969    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921972    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921975    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921977    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921980    2560 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921982    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921985    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:01:22.923321 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921987    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:01:22.923784 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921990    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:01:22.923784 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921992    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:01:22.923784 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921995    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:01:22.923784 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.921997    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:01:22.923784 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922000    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:01:22.923784 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922003    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:01:22.923784 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.922009    2560 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:01:22.923784 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922106    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:01:22.923784 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922111    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:01:22.923784 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922114    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:01:22.923784 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922117    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:01:22.923784 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922119    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:01:22.923784 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922123    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:01:22.923784 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922125    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:01:22.923784 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922128    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922130    2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922133    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922136    2560 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922138    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922141    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922143    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922146    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922148    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922151    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922153    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922157    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922160    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922163    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922166    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922168    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922170    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922173    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922175    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922178    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:01:22.924156 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922180    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922183    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922185    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922188    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922191    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922193    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922196    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922198    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922201    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922203    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922206    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922208    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922211    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922213    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922215    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922218    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922221    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922223    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922226    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922229    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:01:22.924664 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922231    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922234    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922236    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922238    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922241    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922243    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922246    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922248    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922250    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922253    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922255    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922258    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922260    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922262    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922265    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922267    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922270    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922274    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922278    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:01:22.925151 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922281    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922284    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922287    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922290    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922292    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922296    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922298    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922301    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922304    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922307    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922310    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922314    2560 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922316    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922319    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922321    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922324    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922326    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922329    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922331    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:01:22.925670 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:22.922334    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:01:22.926142 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.922339    2560 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:01:22.926142 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.922990    2560 server.go:962] "Client rotation is on, will bootstrap in background"
May 26 16:01:22.926142 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.925379    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 26 16:01:22.926251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.926239    2560 server.go:1019] "Starting client certificate rotation"
May 26 16:01:22.926357 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.926340    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 16:01:22.926960 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.926946    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 16:01:22.949860 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.949842    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 16:01:22.954316 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.954293    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 16:01:22.969337 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.969318    2560 log.go:25] "Validated CRI v1 runtime API"
May 26 16:01:22.975044 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.975025    2560 log.go:25] "Validated CRI v1 image API"
May 26 16:01:22.976428 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.976412    2560 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 26 16:01:22.977427 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.977407    2560 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 16:01:22.978849 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.978830    2560 fs.go:135] Filesystem UUIDs: map[26a385e0-5a90-415b-b6ae-281c134df788:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8609129a-4009-4982-a299-b0f8e8f9d982:/dev/nvme0n1p4]
May 26 16:01:22.978918 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.978848    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 26 16:01:22.983151 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.983046    2560 manager.go:217] Machine: {Timestamp:2026-05-26 16:01:22.982555643 +0000 UTC m=+0.406148795 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100539 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23af047db747124c1dd203af17e292 SystemUUID:ec23af04-7db7-4712-4c1d-d203af17e292 BootID:ea355fed-edca-4171-abf9-78a64ef2faf3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:22:e5:9c:1f:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:22:e5:9c:1f:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:ec:f7:fa:a0:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 26 16:01:22.983877 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.983866    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 26 16:01:22.983958 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.983946    2560 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 26 16:01:22.987924 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.987898    2560 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 26 16:01:22.988058 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.987927    2560 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-193.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 26 16:01:22.988105 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.988068    2560 topology_manager.go:138] "Creating topology manager with none policy"
May 26 16:01:22.988105 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.988077    2560 container_manager_linux.go:306] "Creating device plugin manager"
May 26 16:01:22.988105 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.988084    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 16:01:22.988688 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.988678    2560 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 16:01:22.989916 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.989906    2560 state_mem.go:36] "Initialized new in-memory state store"
May 26 16:01:22.990037 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.990029    2560 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 26 16:01:22.992254 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.992244    2560 kubelet.go:491] "Attempting to sync node with API server"
May 26 16:01:22.992294 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.992257    2560 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 26 16:01:22.992294 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.992269    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 26 16:01:22.992294 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.992277    2560 kubelet.go:397] "Adding apiserver pod source"
May 26 16:01:22.992294 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.992285    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 26 16:01:22.993502 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.993489    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 16:01:22.993574 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.993506    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 16:01:22.996590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.996573    2560 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 26 16:01:22.998817 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:22.998803    2560 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 26 16:01:23.000940 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.000926    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 26 16:01:23.000940 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.000944    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 26 16:01:23.001040 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.000951    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 26 16:01:23.001040 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.000958    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 26 16:01:23.001040 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.000972    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 26 16:01:23.001040 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.000980    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 26 16:01:23.001040 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.000985    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 26 16:01:23.001040 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.000991    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 26 16:01:23.001040 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.001008    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 26 16:01:23.001040 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.001014    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 26 16:01:23.001040 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.001022    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 26 16:01:23.001040 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.001029    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 26 16:01:23.001803 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.001792    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 26 16:01:23.001803 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.001803    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 26 16:01:23.003553 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.003515    2560 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 26 16:01:23.003654 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.003588    2560 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-193.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 26 16:01:23.005710 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.005691    2560 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 26 16:01:23.005794 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.005739    2560 server.go:1295] "Started kubelet"
May 26 16:01:23.005882 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.005815    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 26 16:01:23.005935 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.005854    2560 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 26 16:01:23.005935 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.005917    2560 server_v1.go:47] "podresources" method="list" useActivePods=true
May 26 16:01:23.006667 ip-10-0-137-193 systemd[1]: Started Kubernetes Kubelet.
May 26 16:01:23.007152 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.007090    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w6bhz"
May 26 16:01:23.007340 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.007323    2560 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 26 16:01:23.008015 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.008000    2560 server.go:317] "Adding debug handlers to kubelet server"
May 26 16:01:23.008293 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.008278    2560 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-193.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 26 16:01:23.011734 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.010832    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-193.ec2.internal.18b328ee9ac994de  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-193.ec2.internal,UID:ip-10-0-137-193.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-193.ec2.internal,},FirstTimestamp:2026-05-26 16:01:23.005707486 +0000 UTC m=+0.429300638,LastTimestamp:2026-05-26 16:01:23.005707486 +0000 UTC m=+0.429300638,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-193.ec2.internal,}"
May 26 16:01:23.012825 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.012808    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w6bhz"
May 26 16:01:23.014233 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.014214    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 26 16:01:23.014751 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.014731    2560 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 26 16:01:23.014830 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.014816    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 26 16:01:23.015041 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.015021    2560 volume_manager.go:295] "The desired_state_of_world populator starts"
May 26 16:01:23.015041 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.015038    2560 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 26 16:01:23.015191 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.015132    2560 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 26 16:01:23.015191 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.015176    2560 reconstruct.go:97] "Volume reconstruction finished"
May 26 16:01:23.015191 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.015184    2560 reconciler.go:26] "Reconciler: start to sync state"
May 26 16:01:23.015437 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.015385    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-193.ec2.internal\" not found"
May 26 16:01:23.015569 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.015555    2560 factory.go:55] Registering systemd factory
May 26 16:01:23.015615 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.015575    2560 factory.go:223] Registration of the systemd container factory successfully
May 26 16:01:23.015770 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.015759    2560 factory.go:153] Registering CRI-O factory
May 26 16:01:23.015770 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.015771    2560 factory.go:223] Registration of the crio container factory successfully
May 26 16:01:23.015873 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.015807    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 26 16:01:23.015873 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.015819    2560 factory.go:103] Registering Raw factory
May 26 16:01:23.015873 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.015829    2560 manager.go:1196] Started watching for new ooms in manager
May 26 16:01:23.016201 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.016184    2560 manager.go:319] Starting recovery of all containers
May 26 16:01:23.020010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.019994    2560 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:01:23.023298 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.023134    2560 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-193.ec2.internal\" not found" node="ip-10-0-137-193.ec2.internal"
May 26 16:01:23.023361 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.023216    2560 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 26 16:01:23.025776 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.025749    2560 manager.go:324] Recovery completed
May 26 16:01:23.031557 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.031543    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:01:23.033567 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.033472    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:01:23.033567 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.033505    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:01:23.033567 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.033518    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasSufficientPID"
May 26 16:01:23.034012 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.033998    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 26 16:01:23.034012 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.034013    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 26 16:01:23.034126 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.034034    2560 state_mem.go:36] "Initialized new in-memory state store"
May 26 16:01:23.036824 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.036810    2560 policy_none.go:49] "None policy: Start"
May 26 16:01:23.036897 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.036830    2560 memory_manager.go:186] "Starting memorymanager" policy="None"
May 26 16:01:23.036897 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.036843    2560 state_mem.go:35] "Initializing new in-memory state store"
May 26 16:01:23.084997 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.084982    2560 manager.go:341] "Starting Device Plugin manager"
May 26 16:01:23.097303 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.085010    2560 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 26 16:01:23.097303 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.085019    2560 server.go:85] "Starting device plugin registration server"
May 26 16:01:23.097303 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.085202    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
May 26 16:01:23.097303 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.085211    2560 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 26 16:01:23.097303 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.085284    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 26 16:01:23.097303 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.085364    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 26 16:01:23.097303 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.085374    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 26 16:01:23.097303 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.085835    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 26 16:01:23.097303 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.085872    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-193.ec2.internal\" not found"
May 26 16:01:23.165827 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.165774    2560 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 26 16:01:23.165827 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.165799    2560 status_manager.go:230] "Starting to sync pod status with apiserver"
May 26 16:01:23.165827 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.165817    2560 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 26 16:01:23.165827 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.165826    2560 kubelet.go:2451] "Starting kubelet main sync loop"
May 26 16:01:23.166044 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.165881    2560 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 26 16:01:23.170092 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.170076    2560 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:01:23.185803 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.185787    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:01:23.186691 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.186674    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:01:23.186756 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.186700    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:01:23.186756 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.186711    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasSufficientPID"
May 26 16:01:23.186756 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.186737    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-193.ec2.internal"
May 26 16:01:23.195688 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.195672    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-193.ec2.internal"
May 26 16:01:23.195731 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.195691    2560 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-193.ec2.internal\": node \"ip-10-0-137-193.ec2.internal\" not found"
May 26 16:01:23.213464 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.213445    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-193.ec2.internal\" not found"
May 26 16:01:23.266715 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.266693    2560 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-193.ec2.internal"]
May 26 16:01:23.266771 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.266766    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:01:23.267532 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.267518    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:01:23.267591 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.267545    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:01:23.267591 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.267582    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasSufficientPID"
May 26 16:01:23.268752 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.268741    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:01:23.268917 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.268903    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal"
May 26 16:01:23.268952 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.268933    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:01:23.269385 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.269363    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:01:23.269385 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.269377    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:01:23.269502 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.269386    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:01:23.269502 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.269410    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:01:23.269502 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.269415    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasSufficientPID"
May 26 16:01:23.269502 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.269420    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasSufficientPID"
May 26 16:01:23.270502 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.270489    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-193.ec2.internal"
May 26 16:01:23.270557 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.270511    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:01:23.271107 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.271092    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:01:23.271157 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.271123    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:01:23.271157 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.271137    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeHasSufficientPID"
May 26 16:01:23.303888 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.303865    2560 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-193.ec2.internal\" not found" node="ip-10-0-137-193.ec2.internal"
May 26 16:01:23.307974 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.307961    2560 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-193.ec2.internal\" not found" node="ip-10-0-137-193.ec2.internal"
May 26 16:01:23.314010 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.313996    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-193.ec2.internal\" not found"
May 26 16:01:23.317267 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.317252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a9b584bda4f897e7be9cad1c70cfd55b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal\" (UID: \"a9b584bda4f897e7be9cad1c70cfd55b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal"
May 26 16:01:23.317339 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.317282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b584bda4f897e7be9cad1c70cfd55b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal\" (UID: \"a9b584bda4f897e7be9cad1c70cfd55b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal"
May 26 16:01:23.317339 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.317309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e207af1d280d5e1f7a4ab2b460263c14-config\") pod \"kube-apiserver-proxy-ip-10-0-137-193.ec2.internal\" (UID: \"e207af1d280d5e1f7a4ab2b460263c14\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-193.ec2.internal"
May 26 16:01:23.414984 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.414967    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-193.ec2.internal\" not found"
May 26 16:01:23.418264 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.418229    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e207af1d280d5e1f7a4ab2b460263c14-config\") pod \"kube-apiserver-proxy-ip-10-0-137-193.ec2.internal\" (UID: \"e207af1d280d5e1f7a4ab2b460263c14\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-193.ec2.internal"
May 26 16:01:23.418264 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.418251    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a9b584bda4f897e7be9cad1c70cfd55b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal\" (UID: \"a9b584bda4f897e7be9cad1c70cfd55b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal"
May 26 16:01:23.418379 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.418268    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b584bda4f897e7be9cad1c70cfd55b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal\" (UID: \"a9b584bda4f897e7be9cad1c70cfd55b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal"
May 26 16:01:23.418379 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.418312    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b584bda4f897e7be9cad1c70cfd55b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal\" (UID: \"a9b584bda4f897e7be9cad1c70cfd55b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal"
May 26 16:01:23.418379 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.418335    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e207af1d280d5e1f7a4ab2b460263c14-config\") pod \"kube-apiserver-proxy-ip-10-0-137-193.ec2.internal\" (UID: \"e207af1d280d5e1f7a4ab2b460263c14\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-193.ec2.internal"
May 26 16:01:23.418379 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.418358    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a9b584bda4f897e7be9cad1c70cfd55b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal\" (UID: \"a9b584bda4f897e7be9cad1c70cfd55b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal"
May 26 16:01:23.515478 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.515447    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-193.ec2.internal\" not found"
May 26 16:01:23.607797 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.607776    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal"
May 26 16:01:23.611798 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.611781    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-193.ec2.internal"
May 26 16:01:23.616519 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.616502    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-193.ec2.internal\" not found"
May 26 16:01:23.717046 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.716950    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-193.ec2.internal\" not found"
May 26 16:01:23.817462 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.817422    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-193.ec2.internal\" not found"
May 26 16:01:23.918029 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:23.918003    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-193.ec2.internal\" not found"
May 26 16:01:23.926316 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.926287    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 26 16:01:23.926502 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.926485    2560 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 16:01:23.926566 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:23.926485    2560 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 16:01:24.014476 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:24.014447    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 26 16:01:24.015439 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:24.015408    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-25 15:56:23 +0000 UTC" deadline="2027-12-31 00:38:06.508214956 +0000 UTC"
May 26 16:01:24.015499 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:24.015440    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14000h36m42.492777607s"
May 26 16:01:24.018075 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:24.018059    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-193.ec2.internal\" not found"
May 26 16:01:24.027582 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:24.027563    2560 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 16:01:24.081891 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:24.081864    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n2xf5"
May 26 16:01:24.088025 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:24.088008    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n2xf5"
May 26 16:01:24.118367 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:24.118342    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-193.ec2.internal\" not found"
May 26 16:01:24.188351 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:24.188334    2560 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:01:24.199037 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:24.199005    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b584bda4f897e7be9cad1c70cfd55b.slice/crio-49f199dd6e5ae1dfcf4b86d03f031b9c11c10a674f251e41fcce9a29e570475b WatchSource:0}: Error finding container 49f199dd6e5ae1dfcf4b86d03f031b9c11c10a674f251e41fcce9a29e570475b: Status 404 returned error can't find the container with id 49f199dd6e5ae1dfcf4b86d03f031b9c11c10a674f251e41fcce9a29e570475b
May 26 16:01:24.199240 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:24.199219    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode207af1d280d5e1f7a4ab2b460263c14.slice/crio-8adf165f08b3aa9daadf0890d2b43efe64ec00737402d2317cf8ea9b9037c4bd WatchSource:0}: Error finding container 8adf165f08b3aa9daadf0890d2b43efe64ec00737402d2317cf8ea9b9037c4bd: Status 404 returned error can't find the container with id 8adf165f08b3aa9daadf0890d2b43efe64ec00737402d2317cf8ea9b9037c4bd
May 26 16:01:24.203656 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:24.203640    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:01:24.219172 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:24.219147    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-193.ec2.internal\" not found"
May 26 16:01:24.269738 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:24.269661    2560 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:01:24.315474 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:24.315449    2560 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal"
May 26 16:01:24.329283 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:24.329265    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 16:01:24.330334 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:24.330321    2560 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-193.ec2.internal"
May 26 16:01:24.336585 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:24.336568    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 16:01:24.993239 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:24.993205    2560 apiserver.go:52] "Watching apiserver"
May 26 16:01:25.000508 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.000479    2560 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 26 16:01:25.002620 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.002596    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-dd7q4","openshift-ovn-kubernetes/ovnkube-node-n84fp","kube-system/konnectivity-agent-t2rzl","openshift-image-registry/node-ca-mz8tq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal","openshift-multus/multus-additional-cni-plugins-vw726","openshift-network-operator/iptables-alerter-nnztf","kube-system/kube-apiserver-proxy-ip-10-0-137-193.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb","openshift-cluster-node-tuning-operator/tuned-zgpnk","openshift-dns/node-resolver-xx9vq","openshift-multus/multus-rg5xr","openshift-multus/network-metrics-daemon-j528d"]
May 26 16:01:25.004076 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.004057    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nnztf"
May 26 16:01:25.008256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.006969    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.008256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.007160    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dr2pn\""
May 26 16:01:25.008256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.007471    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 26 16:01:25.008256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.007472    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 26 16:01:25.008256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.008058    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 26 16:01:25.009508 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.009284    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 26 16:01:25.011029 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.010782    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 26 16:01:25.011029 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.010800    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 26 16:01:25.011029 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.010822    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 26 16:01:25.011029 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.010828    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 26 16:01:25.011029 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.010782    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 26 16:01:25.011029 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.010959    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qz8wz\""
May 26 16:01:25.011994 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.011962    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t2rzl"
May 26 16:01:25.013538 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.013382    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mz8tq"
May 26 16:01:25.013641 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.013543    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.014076 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.014056    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 26 16:01:25.014155 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.014104    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p72cf\""
May 26 16:01:25.014434 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.014417    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 26 16:01:25.014798 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.014778    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:25.014909 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:25.014884    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:25.016162 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.016135    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 26 16:01:25.016266 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.016179    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 26 16:01:25.016266 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.016213    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 26 16:01:25.016266 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.016247    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8tnc4\""
May 26 16:01:25.016535 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.016517    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 26 16:01:25.016741 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.016725    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6d4qv\""
May 26 16:01:25.016863 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.016845    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 26 16:01:25.017000 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.016981    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 26 16:01:25.017097 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.017080    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 26 16:01:25.017202 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.017185    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 26 16:01:25.018706 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.018423    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.018706 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.018460    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.020056 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.019968    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xx9vq"
May 26 16:01:25.020588 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.020570    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 26 16:01:25.020792 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.020768    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h8xtp\""
May 26 16:01:25.020792 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.020787    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 26 16:01:25.020936 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.020844    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 26 16:01:25.021673 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.021653    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 26 16:01:25.021938 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.021922    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k5f4l\""
May 26 16:01:25.022599 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.022515    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 26 16:01:25.022599 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.022548    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 26 16:01:25.022599 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.022570    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 26 16:01:25.022910 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.022890    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hqbdt\""
May 26 16:01:25.023600 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.023577    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:25.023684 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:25.023654    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:25.023684 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.023679    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.025749 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.025730    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 26 16:01:25.025946 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.025898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.026024 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.025988    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-socket-dir\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.026080 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026021    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea5226b3-f441-403d-aa44-8ac33afd311b-serviceca\") pod \"node-ca-mz8tq\" (UID: \"ea5226b3-f441-403d-aa44-8ac33afd311b\") " pod="openshift-image-registry/node-ca-mz8tq"
May 26 16:01:25.026080 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026048    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-system-cni-dir\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.026182 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026086    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.026182 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026111    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1920789f-1573-4118-a9a4-7e3a44fcb1ad-iptables-alerter-script\") pod \"iptables-alerter-nnztf\" (UID: \"1920789f-1573-4118-a9a4-7e3a44fcb1ad\") " pod="openshift-network-operator/iptables-alerter-nnztf"
May 26 16:01:25.026182 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026134    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-registration-dir\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.026182 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026164    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-run-ovn\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.026379 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026187    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-node-log\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.026379 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aaf93059-5a36-41c9-901e-fb7ae0e4224f-konnectivity-ca\") pod \"konnectivity-agent-t2rzl\" (UID: \"aaf93059-5a36-41c9-901e-fb7ae0e4224f\") " pod="kube-system/konnectivity-agent-t2rzl"
May 26 16:01:25.026379 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-cnibin\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.026379 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk6d6\" (UniqueName: \"kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6\") pod \"network-check-target-dd7q4\" (UID: \"67ccb020-7afc-463c-a316-263722c743ad\") " pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:25.026379 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1920789f-1573-4118-a9a4-7e3a44fcb1ad-host-slash\") pod \"iptables-alerter-nnztf\" (UID: \"1920789f-1573-4118-a9a4-7e3a44fcb1ad\") " pod="openshift-network-operator/iptables-alerter-nnztf"
May 26 16:01:25.026379 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026343    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.026379 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.026705 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026408    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbswp\" (UniqueName: \"kubernetes.io/projected/26ce8382-d0aa-4fe6-8a61-7df596a1264f-kube-api-access-vbswp\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.026705 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026440    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-kubelet\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.026705 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026463    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/078ebf89-62fd-47e8-8a9f-03bf7431c882-ovnkube-config\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.026705 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026490    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-device-dir\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.026705 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026513    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-etc-openvswitch\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.026705 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-run-ovn-kubernetes\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.026705 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026617    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-cni-netd\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.026705 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026646    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/078ebf89-62fd-47e8-8a9f-03bf7431c882-ovn-node-metrics-cert\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.027071 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026708    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.027071 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026740    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-systemd-units\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.027071 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026766    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-run-systemd\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.027071 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-var-lib-openvswitch\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.027071 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026852    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kfhvp\""
May 26 16:01:25.027071 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026876    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-run-openvswitch\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.027071 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026901    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/078ebf89-62fd-47e8-8a9f-03bf7431c882-env-overrides\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.027071 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-cni-binary-copy\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.027071 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.026979    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-sys-fs\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.027071 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.027023    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-cni-bin\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.027071 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.027046    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.027590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.027085    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/078ebf89-62fd-47e8-8a9f-03bf7431c882-ovnkube-script-lib\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.027590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.027122    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27vg7\" (UniqueName: \"kubernetes.io/projected/078ebf89-62fd-47e8-8a9f-03bf7431c882-kube-api-access-27vg7\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.027590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.027163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aaf93059-5a36-41c9-901e-fb7ae0e4224f-agent-certs\") pod \"konnectivity-agent-t2rzl\" (UID: \"aaf93059-5a36-41c9-901e-fb7ae0e4224f\") " pod="kube-system/konnectivity-agent-t2rzl"
May 26 16:01:25.027590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.027209    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea5226b3-f441-403d-aa44-8ac33afd311b-host\") pod \"node-ca-mz8tq\" (UID: \"ea5226b3-f441-403d-aa44-8ac33afd311b\") " pod="openshift-image-registry/node-ca-mz8tq"
May 26 16:01:25.027590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.027253    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhwws\" (UniqueName: \"kubernetes.io/projected/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-kube-api-access-rhwws\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.027590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.027289    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjrmc\" (UniqueName: \"kubernetes.io/projected/1920789f-1573-4118-a9a4-7e3a44fcb1ad-kube-api-access-tjrmc\") pod \"iptables-alerter-nnztf\" (UID: \"1920789f-1573-4118-a9a4-7e3a44fcb1ad\") " pod="openshift-network-operator/iptables-alerter-nnztf"
May 26 16:01:25.027590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.027331    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-slash\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.027590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.027370    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-log-socket\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.027590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.027441    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqnds\" (UniqueName: \"kubernetes.io/projected/ea5226b3-f441-403d-aa44-8ac33afd311b-kube-api-access-qqnds\") pod \"node-ca-mz8tq\" (UID: \"ea5226b3-f441-403d-aa44-8ac33afd311b\") " pod="openshift-image-registry/node-ca-mz8tq"
May 26 16:01:25.027590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.027467    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-os-release\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.027590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.027505    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-run-netns\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.089108 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.089072    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 15:56:24 +0000 UTC" deadline="2028-02-25 04:08:30.49716208 +0000 UTC"
May 26 16:01:25.089108 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.089107    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15348h7m5.408058389s"
May 26 16:01:25.116494 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.116459    2560 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 26 16:01:25.128444 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128415    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-cnibin\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.128569 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128457    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1920789f-1573-4118-a9a4-7e3a44fcb1ad-host-slash\") pod \"iptables-alerter-nnztf\" (UID: \"1920789f-1573-4118-a9a4-7e3a44fcb1ad\") " pod="openshift-network-operator/iptables-alerter-nnztf"
May 26 16:01:25.128569 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128483    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.128569 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128508    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-kubelet\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.128569 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128531    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/078ebf89-62fd-47e8-8a9f-03bf7431c882-ovnkube-config\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.128569 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128560    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-tuned\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.128825 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128585    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-device-dir\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.128825 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128607    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-cni-netd\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.128825 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128632    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.128825 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128658    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-run-systemd\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.128825 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128680    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-run-openvswitch\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.128825 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128703    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/078ebf89-62fd-47e8-8a9f-03bf7431c882-env-overrides\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.128825 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128731    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-sysconfig\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.128825 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128756    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-sysctl-d\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.128825 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128782    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-cni-binary-copy\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.128825 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128808    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc50ed1b-7010-4636-b832-6fd7c64b57da-tmp\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.129195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128833    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1e36e32-7353-4703-8b16-848a49a47e91-hosts-file\") pod \"node-resolver-xx9vq\" (UID: \"a1e36e32-7353-4703-8b16-848a49a47e91\") " pod="openshift-dns/node-resolver-xx9vq"
May 26 16:01:25.129195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128862    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrmc\" (UniqueName: \"kubernetes.io/projected/1920789f-1573-4118-a9a4-7e3a44fcb1ad-kube-api-access-tjrmc\") pod \"iptables-alerter-nnztf\" (UID: \"1920789f-1573-4118-a9a4-7e3a44fcb1ad\") " pod="openshift-network-operator/iptables-alerter-nnztf"
May 26 16:01:25.129195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128888    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-log-socket\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.129195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128914    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qqnds\" (UniqueName: \"kubernetes.io/projected/ea5226b3-f441-403d-aa44-8ac33afd311b-kube-api-access-qqnds\") pod \"node-ca-mz8tq\" (UID: \"ea5226b3-f441-403d-aa44-8ac33afd311b\") " pod="openshift-image-registry/node-ca-mz8tq"
May 26 16:01:25.129195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-sysctl-conf\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.129195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128968    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-multus-cni-dir\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.129195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.128993    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-cnibin\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.129195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-var-lib-kubelet\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.129195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129040    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-multus-conf-dir\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.129195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129065    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-run-netns\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.129195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129088    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-host\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.129195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129113    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-os-release\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.129195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129137    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-var-lib-cni-bin\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.129195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129161    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0f99d273-701a-4186-836a-333aa040304b-multus-daemon-config\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.129195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129187    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-run-multus-certs\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.129807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129213    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx6pd\" (UniqueName: \"kubernetes.io/projected/0f99d273-701a-4186-836a-333aa040304b-kube-api-access-dx6pd\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.129807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129242    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.129807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129274    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-socket-dir\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.129807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129301    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea5226b3-f441-403d-aa44-8ac33afd311b-serviceca\") pod \"node-ca-mz8tq\" (UID: \"ea5226b3-f441-403d-aa44-8ac33afd311b\") " pod="openshift-image-registry/node-ca-mz8tq"
May 26 16:01:25.129807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129330    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-registration-dir\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.129807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129356    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-run-ovn\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.129807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129379    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-node-log\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.129807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129419    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aaf93059-5a36-41c9-901e-fb7ae0e4224f-konnectivity-ca\") pod \"konnectivity-agent-t2rzl\" (UID: \"aaf93059-5a36-41c9-901e-fb7ae0e4224f\") " pod="kube-system/konnectivity-agent-t2rzl"
May 26 16:01:25.129807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129445    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-modprobe-d\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.129807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129524    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-run-ovn\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.129807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129589    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-node-log\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.129807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129669    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-socket-dir\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.129807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129687    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-registration-dir\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.129807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129734    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-run-netns\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.129807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129783    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-cnibin\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.129807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129796    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-cni-netd\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129826    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129863    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-kubelet\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.129987    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea5226b3-f441-403d-aa44-8ac33afd311b-serviceca\") pod \"node-ca-mz8tq\" (UID: \"ea5226b3-f441-403d-aa44-8ac33afd311b\") " pod="openshift-image-registry/node-ca-mz8tq"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-run\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130073    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1920789f-1573-4118-a9a4-7e3a44fcb1ad-host-slash\") pod \"iptables-alerter-nnztf\" (UID: \"1920789f-1573-4118-a9a4-7e3a44fcb1ad\") " pod="openshift-network-operator/iptables-alerter-nnztf"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130095    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aaf93059-5a36-41c9-901e-fb7ae0e4224f-konnectivity-ca\") pod \"konnectivity-agent-t2rzl\" (UID: \"aaf93059-5a36-41c9-901e-fb7ae0e4224f\") " pod="kube-system/konnectivity-agent-t2rzl"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130206    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-log-socket\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130243    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-device-dir\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130290    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-run-openvswitch\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130321    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-cni-binary-copy\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130339    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/078ebf89-62fd-47e8-8a9f-03bf7431c882-ovnkube-config\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130373    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6d6\" (UniqueName: \"kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6\") pod \"network-check-target-dd7q4\" (UID: \"67ccb020-7afc-463c-a316-263722c743ad\") " pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130410    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-run-systemd\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130418    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130433    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130475    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vbswp\" (UniqueName: \"kubernetes.io/projected/26ce8382-d0aa-4fe6-8a61-7df596a1264f-kube-api-access-vbswp\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.130607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130517    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f99d273-701a-4186-836a-333aa040304b-cni-binary-copy\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130629    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-run-k8s-cni-cncf-io\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-run-netns\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130707    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-etc-kubernetes\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130728    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130728    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130756    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-etc-openvswitch\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130777    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/078ebf89-62fd-47e8-8a9f-03bf7431c882-env-overrides\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130797    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-run-ovn-kubernetes\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130807    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-etc-openvswitch\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130867    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-run-ovn-kubernetes\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130857    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/078ebf89-62fd-47e8-8a9f-03bf7431c882-ovn-node-metrics-cert\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.130934    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs\") pod \"network-metrics-daemon-j528d\" (UID: \"22219f09-c007-46a5-b425-df729845bf45\") " pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131031    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-multus-socket-dir-parent\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131074    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-systemd-units\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131103    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-var-lib-openvswitch\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131146    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/078ebf89-62fd-47e8-8a9f-03bf7431c882-ovnkube-script-lib\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.131256 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131174    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-27vg7\" (UniqueName: \"kubernetes.io/projected/078ebf89-62fd-47e8-8a9f-03bf7431c882-kube-api-access-27vg7\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131206    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-var-lib-openvswitch\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131208    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131255    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1e36e32-7353-4703-8b16-848a49a47e91-tmp-dir\") pod \"node-resolver-xx9vq\" (UID: \"a1e36e32-7353-4703-8b16-848a49a47e91\") " pod="openshift-dns/node-resolver-xx9vq"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131284    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-sys-fs\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131297    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-systemd-units\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131435    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-cni-bin\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131466    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131495    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aaf93059-5a36-41c9-901e-fb7ae0e4224f-agent-certs\") pod \"konnectivity-agent-t2rzl\" (UID: \"aaf93059-5a36-41c9-901e-fb7ae0e4224f\") " pod="kube-system/konnectivity-agent-t2rzl"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131556    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-cni-bin\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131611    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea5226b3-f441-403d-aa44-8ac33afd311b-host\") pod \"node-ca-mz8tq\" (UID: \"ea5226b3-f441-403d-aa44-8ac33afd311b\") " pod="openshift-image-registry/node-ca-mz8tq"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-systemd\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131712    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzz84\" (UniqueName: \"kubernetes.io/projected/22219f09-c007-46a5-b425-df729845bf45-kube-api-access-tzz84\") pod \"network-metrics-daemon-j528d\" (UID: \"22219f09-c007-46a5-b425-df729845bf45\") " pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131782    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rhwws\" (UniqueName: \"kubernetes.io/projected/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-kube-api-access-rhwws\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131822    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-slash\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-kubernetes\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-955wp\" (UniqueName: \"kubernetes.io/projected/a1e36e32-7353-4703-8b16-848a49a47e91-kube-api-access-955wp\") pod \"node-resolver-xx9vq\" (UID: \"a1e36e32-7353-4703-8b16-848a49a47e91\") " pod="openshift-dns/node-resolver-xx9vq"
May 26 16:01:25.132032 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131905    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-var-lib-cni-multus\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.132806 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131934    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-os-release\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.132806 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131940    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26ce8382-d0aa-4fe6-8a61-7df596a1264f-sys-fs\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.132806 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-sys\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.132806 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131981    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea5226b3-f441-403d-aa44-8ac33afd311b-host\") pod \"node-ca-mz8tq\" (UID: \"ea5226b3-f441-403d-aa44-8ac33afd311b\") " pod="openshift-image-registry/node-ca-mz8tq"
May 26 16:01:25.132806 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.131989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-var-lib-kubelet\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.132806 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.132060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng6cg\" (UniqueName: \"kubernetes.io/projected/fc50ed1b-7010-4636-b832-6fd7c64b57da-kube-api-access-ng6cg\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.132806 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.132091    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-hostroot\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.132806 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.132120    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-system-cni-dir\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.132806 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.132149    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.132806 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.132176    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1920789f-1573-4118-a9a4-7e3a44fcb1ad-iptables-alerter-script\") pod \"iptables-alerter-nnztf\" (UID: \"1920789f-1573-4118-a9a4-7e3a44fcb1ad\") " pod="openshift-network-operator/iptables-alerter-nnztf"
May 26 16:01:25.132806 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.132202    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-lib-modules\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.132806 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.132230    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-system-cni-dir\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.132806 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.132319    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-slash\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.132806 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.132339    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/078ebf89-62fd-47e8-8a9f-03bf7431c882-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.132806 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.132366    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-system-cni-dir\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.133486 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.132841    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1920789f-1573-4118-a9a4-7e3a44fcb1ad-iptables-alerter-script\") pod \"iptables-alerter-nnztf\" (UID: \"1920789f-1573-4118-a9a4-7e3a44fcb1ad\") " pod="openshift-network-operator/iptables-alerter-nnztf"
May 26 16:01:25.133486 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.132879    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/078ebf89-62fd-47e8-8a9f-03bf7431c882-ovnkube-script-lib\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.133486 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.133139    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-os-release\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.133627 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.133592    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.134585 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.134561    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/078ebf89-62fd-47e8-8a9f-03bf7431c882-ovn-node-metrics-cert\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.134705 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.134637    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aaf93059-5a36-41c9-901e-fb7ae0e4224f-agent-certs\") pod \"konnectivity-agent-t2rzl\" (UID: \"aaf93059-5a36-41c9-901e-fb7ae0e4224f\") " pod="kube-system/konnectivity-agent-t2rzl"
May 26 16:01:25.135935 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:25.135621    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:25.135935 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:25.135642    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:25.135935 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:25.135654    2560 projected.go:194] Error preparing data for projected volume kube-api-access-rk6d6 for pod openshift-network-diagnostics/network-check-target-dd7q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:25.135935 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:25.135745    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6 podName:67ccb020-7afc-463c-a316-263722c743ad nodeName:}" failed. No retries permitted until 2026-05-26 16:01:25.635711553 +0000 UTC m=+3.059304713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rk6d6" (UniqueName: "kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6") pod "network-check-target-dd7q4" (UID: "67ccb020-7afc-463c-a316-263722c743ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:25.137651 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.137574    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrmc\" (UniqueName: \"kubernetes.io/projected/1920789f-1573-4118-a9a4-7e3a44fcb1ad-kube-api-access-tjrmc\") pod \"iptables-alerter-nnztf\" (UID: \"1920789f-1573-4118-a9a4-7e3a44fcb1ad\") " pod="openshift-network-operator/iptables-alerter-nnztf"
May 26 16:01:25.137651 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.137620    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqnds\" (UniqueName: \"kubernetes.io/projected/ea5226b3-f441-403d-aa44-8ac33afd311b-kube-api-access-qqnds\") pod \"node-ca-mz8tq\" (UID: \"ea5226b3-f441-403d-aa44-8ac33afd311b\") " pod="openshift-image-registry/node-ca-mz8tq"
May 26 16:01:25.142526 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.142504    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27vg7\" (UniqueName: \"kubernetes.io/projected/078ebf89-62fd-47e8-8a9f-03bf7431c882-kube-api-access-27vg7\") pod \"ovnkube-node-n84fp\" (UID: \"078ebf89-62fd-47e8-8a9f-03bf7431c882\") " pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.142963 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.142920    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbswp\" (UniqueName: \"kubernetes.io/projected/26ce8382-d0aa-4fe6-8a61-7df596a1264f-kube-api-access-vbswp\") pod \"aws-ebs-csi-driver-node-mxgzb\" (UID: \"26ce8382-d0aa-4fe6-8a61-7df596a1264f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.143749 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.143711    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhwws\" (UniqueName: \"kubernetes.io/projected/1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6-kube-api-access-rhwws\") pod \"multus-additional-cni-plugins-vw726\" (UID: \"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6\") " pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.172850 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.172802    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-193.ec2.internal" event={"ID":"e207af1d280d5e1f7a4ab2b460263c14","Type":"ContainerStarted","Data":"8adf165f08b3aa9daadf0890d2b43efe64ec00737402d2317cf8ea9b9037c4bd"}
May 26 16:01:25.174382 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.174265    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal" event={"ID":"a9b584bda4f897e7be9cad1c70cfd55b","Type":"ContainerStarted","Data":"49f199dd6e5ae1dfcf4b86d03f031b9c11c10a674f251e41fcce9a29e570475b"}
May 26 16:01:25.233403 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233356    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-var-lib-kubelet\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.233545 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233414    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ng6cg\" (UniqueName: \"kubernetes.io/projected/fc50ed1b-7010-4636-b832-6fd7c64b57da-kube-api-access-ng6cg\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.233545 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233440    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-hostroot\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.233545 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233467    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-lib-modules\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.233545 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233490    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-system-cni-dir\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.233545 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233517    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-tuned\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.233545 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233546    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-sysconfig\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.233793 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233568    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-sysctl-d\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.233793 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233594    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc50ed1b-7010-4636-b832-6fd7c64b57da-tmp\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.233793 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233617    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1e36e32-7353-4703-8b16-848a49a47e91-hosts-file\") pod \"node-resolver-xx9vq\" (UID: \"a1e36e32-7353-4703-8b16-848a49a47e91\") " pod="openshift-dns/node-resolver-xx9vq"
May 26 16:01:25.233793 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233642    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-sysctl-conf\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.233793 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233648    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-hostroot\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.233793 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233664    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-multus-cni-dir\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.233793 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233706    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-sysconfig\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.233793 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233708    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-system-cni-dir\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.233793 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233722    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-multus-cni-dir\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.233793 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233486    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-var-lib-kubelet\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.233793 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233738    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-cnibin\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.233793 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233769    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-var-lib-kubelet\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233810    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-multus-conf-dir\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233835    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-sysctl-d\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233836    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-host\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233872    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-os-release\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233878    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-host\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233897    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-var-lib-cni-bin\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233924    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0f99d273-701a-4186-836a-333aa040304b-multus-daemon-config\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233926    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-cnibin\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233948    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-run-multus-certs\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233959    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-var-lib-kubelet\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233972    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dx6pd\" (UniqueName: \"kubernetes.io/projected/0f99d273-701a-4186-836a-333aa040304b-kube-api-access-dx6pd\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234000    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-multus-conf-dir\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234010    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-modprobe-d\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234036    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-run\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234061    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-os-release\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234072    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f99d273-701a-4186-836a-333aa040304b-cni-binary-copy\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234117    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-var-lib-cni-bin\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.234251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234153    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-run-k8s-cni-cncf-io\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234181    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-run-netns\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234205    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-etc-kubernetes\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234233    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs\") pod \"network-metrics-daemon-j528d\" (UID: \"22219f09-c007-46a5-b425-df729845bf45\") " pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234258    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-multus-socket-dir-parent\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234285    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1e36e32-7353-4703-8b16-848a49a47e91-tmp-dir\") pod \"node-resolver-xx9vq\" (UID: \"a1e36e32-7353-4703-8b16-848a49a47e91\") " pod="openshift-dns/node-resolver-xx9vq"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234327    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-systemd\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234356    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tzz84\" (UniqueName: \"kubernetes.io/projected/22219f09-c007-46a5-b425-df729845bf45-kube-api-access-tzz84\") pod \"network-metrics-daemon-j528d\" (UID: \"22219f09-c007-46a5-b425-df729845bf45\") " pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234429    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-kubernetes\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.233616    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-lib-modules\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234456    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-955wp\" (UniqueName: \"kubernetes.io/projected/a1e36e32-7353-4703-8b16-848a49a47e91-kube-api-access-955wp\") pod \"node-resolver-xx9vq\" (UID: \"a1e36e32-7353-4703-8b16-848a49a47e91\") " pod="openshift-dns/node-resolver-xx9vq"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234484    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-var-lib-cni-multus\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234510    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-sys\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234576    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1e36e32-7353-4703-8b16-848a49a47e91-hosts-file\") pod \"node-resolver-xx9vq\" (UID: \"a1e36e32-7353-4703-8b16-848a49a47e91\") " pod="openshift-dns/node-resolver-xx9vq"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234588    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-sys\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234613    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f99d273-701a-4186-836a-333aa040304b-cni-binary-copy\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234671    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-run\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234708    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-modprobe-d\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.235010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234941    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-run-netns\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.235583 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234942    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-multus-socket-dir-parent\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.235583 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234974    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-etc-kubernetes\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.235583 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234999    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0f99d273-701a-4186-836a-333aa040304b-multus-daemon-config\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.235583 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.234994    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-run-multus-certs\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.235583 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.235040    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-run-k8s-cni-cncf-io\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.235583 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:25.235060    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:25.235583 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:25.235115    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs podName:22219f09-c007-46a5-b425-df729845bf45 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:25.735097518 +0000 UTC m=+3.158690661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs") pod "network-metrics-daemon-j528d" (UID: "22219f09-c007-46a5-b425-df729845bf45") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:25.235583 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.235179    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-sysctl-conf\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.235583 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.235234    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-kubernetes\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.235583 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.235234    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1e36e32-7353-4703-8b16-848a49a47e91-tmp-dir\") pod \"node-resolver-xx9vq\" (UID: \"a1e36e32-7353-4703-8b16-848a49a47e91\") " pod="openshift-dns/node-resolver-xx9vq"
May 26 16:01:25.235583 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.235284    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0f99d273-701a-4186-836a-333aa040304b-host-var-lib-cni-multus\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.235583 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.235314    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-systemd\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.236557 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.236534    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc50ed1b-7010-4636-b832-6fd7c64b57da-tmp\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.237708 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.237684    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc50ed1b-7010-4636-b832-6fd7c64b57da-etc-tuned\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.248258 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.248202    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng6cg\" (UniqueName: \"kubernetes.io/projected/fc50ed1b-7010-4636-b832-6fd7c64b57da-kube-api-access-ng6cg\") pod \"tuned-zgpnk\" (UID: \"fc50ed1b-7010-4636-b832-6fd7c64b57da\") " pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.249325 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.249302    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-955wp\" (UniqueName: \"kubernetes.io/projected/a1e36e32-7353-4703-8b16-848a49a47e91-kube-api-access-955wp\") pod \"node-resolver-xx9vq\" (UID: \"a1e36e32-7353-4703-8b16-848a49a47e91\") " pod="openshift-dns/node-resolver-xx9vq"
May 26 16:01:25.249600 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.249578    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx6pd\" (UniqueName: \"kubernetes.io/projected/0f99d273-701a-4186-836a-333aa040304b-kube-api-access-dx6pd\") pod \"multus-rg5xr\" (UID: \"0f99d273-701a-4186-836a-333aa040304b\") " pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.250552 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.250498    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzz84\" (UniqueName: \"kubernetes.io/projected/22219f09-c007-46a5-b425-df729845bf45-kube-api-access-tzz84\") pod \"network-metrics-daemon-j528d\" (UID: \"22219f09-c007-46a5-b425-df729845bf45\") " pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:25.318399 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.318355    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nnztf"
May 26 16:01:25.327271 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.327248    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:25.336910 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.336889    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t2rzl"
May 26 16:01:25.344463 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.344444    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mz8tq"
May 26 16:01:25.346674 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.346660    2560 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:01:25.352481 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.352466    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vw726"
May 26 16:01:25.360059 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.360044    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb"
May 26 16:01:25.368649 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.368631    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zgpnk"
May 26 16:01:25.381161 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.381144    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xx9vq"
May 26 16:01:25.386713 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.386693    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rg5xr"
May 26 16:01:25.516994 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.516917    2560 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:01:25.637173 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.637139    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6d6\" (UniqueName: \"kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6\") pod \"network-check-target-dd7q4\" (UID: \"67ccb020-7afc-463c-a316-263722c743ad\") " pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:25.637335 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:25.637315    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:25.637408 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:25.637341    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:25.637408 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:25.637351    2560 projected.go:194] Error preparing data for projected volume kube-api-access-rk6d6 for pod openshift-network-diagnostics/network-check-target-dd7q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:25.637507 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:25.637424    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6 podName:67ccb020-7afc-463c-a316-263722c743ad nodeName:}" failed. No retries permitted until 2026-05-26 16:01:26.637404127 +0000 UTC m=+4.060997283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rk6d6" (UniqueName: "kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6") pod "network-check-target-dd7q4" (UID: "67ccb020-7afc-463c-a316-263722c743ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:25.738111 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:25.738078    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs\") pod \"network-metrics-daemon-j528d\" (UID: \"22219f09-c007-46a5-b425-df729845bf45\") " pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:25.738259 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:25.738208    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:25.738332 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:25.738278    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs podName:22219f09-c007-46a5-b425-df729845bf45 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:26.738259691 +0000 UTC m=+4.161852833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs") pod "network-metrics-daemon-j528d" (UID: "22219f09-c007-46a5-b425-df729845bf45") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:25.923074 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:25.923036    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078ebf89_62fd_47e8_8a9f_03bf7431c882.slice/crio-3500e1bfa0fdc03348a12f1b65d0a04b9d4f55acad55c154202deb3bd6954f4f WatchSource:0}: Error finding container 3500e1bfa0fdc03348a12f1b65d0a04b9d4f55acad55c154202deb3bd6954f4f: Status 404 returned error can't find the container with id 3500e1bfa0fdc03348a12f1b65d0a04b9d4f55acad55c154202deb3bd6954f4f
May 26 16:01:25.923833 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:25.923795    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f99d273_701a_4186_836a_333aa040304b.slice/crio-0df4d75467b90c9f2d10524cb30f7aa62d28b9beaeab59fc40924778499aa9c6 WatchSource:0}: Error finding container 0df4d75467b90c9f2d10524cb30f7aa62d28b9beaeab59fc40924778499aa9c6: Status 404 returned error can't find the container with id 0df4d75467b90c9f2d10524cb30f7aa62d28b9beaeab59fc40924778499aa9c6
May 26 16:01:25.924879 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:25.924846    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ce8382_d0aa_4fe6_8a61_7df596a1264f.slice/crio-1ef488103aa56fcce79c9847d338de2fd3c4b1e4f881509da599a7eb6d39fb7a WatchSource:0}: Error finding container 1ef488103aa56fcce79c9847d338de2fd3c4b1e4f881509da599a7eb6d39fb7a: Status 404 returned error can't find the container with id 1ef488103aa56fcce79c9847d338de2fd3c4b1e4f881509da599a7eb6d39fb7a
May 26 16:01:25.925788 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:25.925731    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1920789f_1573_4118_a9a4_7e3a44fcb1ad.slice/crio-839f18b62970b7bb3f3e41211cc86b5baebffb608e2dbce977beb39d23112df7 WatchSource:0}: Error finding container 839f18b62970b7bb3f3e41211cc86b5baebffb608e2dbce977beb39d23112df7: Status 404 returned error can't find the container with id 839f18b62970b7bb3f3e41211cc86b5baebffb608e2dbce977beb39d23112df7
May 26 16:01:25.930531 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:25.930502    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1556a0fe_3c5b_4bcc_97e3_f5c361a44bd6.slice/crio-a67cbe5bf73b891178920f139089f76af936b6b06b67b71bfadef09d337d0ed5 WatchSource:0}: Error finding container a67cbe5bf73b891178920f139089f76af936b6b06b67b71bfadef09d337d0ed5: Status 404 returned error can't find the container with id a67cbe5bf73b891178920f139089f76af936b6b06b67b71bfadef09d337d0ed5
May 26 16:01:25.931360 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:25.931341    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e36e32_7353_4703_8b16_848a49a47e91.slice/crio-59667012c3a7e2e726c973316f7d4c59ec815ebe70325f7d015b2cb51a050623 WatchSource:0}: Error finding container 59667012c3a7e2e726c973316f7d4c59ec815ebe70325f7d015b2cb51a050623: Status 404 returned error can't find the container with id 59667012c3a7e2e726c973316f7d4c59ec815ebe70325f7d015b2cb51a050623
May 26 16:01:25.933004 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:25.932715    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5226b3_f441_403d_aa44_8ac33afd311b.slice/crio-d3080fdc7d78bed1cec7e52f85c4c40aec8df6e1eea86614eb9f2b6269c700f3 WatchSource:0}: Error finding container d3080fdc7d78bed1cec7e52f85c4c40aec8df6e1eea86614eb9f2b6269c700f3: Status 404 returned error can't find the container with id d3080fdc7d78bed1cec7e52f85c4c40aec8df6e1eea86614eb9f2b6269c700f3
May 26 16:01:25.933150 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:25.933094    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc50ed1b_7010_4636_b832_6fd7c64b57da.slice/crio-a0cfd2f70d3a3bd174d271809030bd88c70d5033525aa4852fda20b8dee09655 WatchSource:0}: Error finding container a0cfd2f70d3a3bd174d271809030bd88c70d5033525aa4852fda20b8dee09655: Status 404 returned error can't find the container with id a0cfd2f70d3a3bd174d271809030bd88c70d5033525aa4852fda20b8dee09655
May 26 16:01:25.934301 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:01:25.934013    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf93059_5a36_41c9_901e_fb7ae0e4224f.slice/crio-4d24632e13209455448c3767e6ade48523e6e6721ddf4275499fa9899f7e1ee5 WatchSource:0}: Error finding container 4d24632e13209455448c3767e6ade48523e6e6721ddf4275499fa9899f7e1ee5: Status 404 returned error can't find the container with id 4d24632e13209455448c3767e6ade48523e6e6721ddf4275499fa9899f7e1ee5
May 26 16:01:26.090436 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:26.090185    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 15:56:24 +0000 UTC" deadline="2028-01-30 14:07:06.587990061 +0000 UTC"
May 26 16:01:26.090436 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:26.090435    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14734h5m40.497560206s"
May 26 16:01:26.166661 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:26.166562    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:26.166790 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:26.166681    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:26.176501 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:26.176477    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xx9vq" event={"ID":"a1e36e32-7353-4703-8b16-848a49a47e91","Type":"ContainerStarted","Data":"59667012c3a7e2e726c973316f7d4c59ec815ebe70325f7d015b2cb51a050623"}
May 26 16:01:26.177405 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:26.177373    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vw726" event={"ID":"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6","Type":"ContainerStarted","Data":"a67cbe5bf73b891178920f139089f76af936b6b06b67b71bfadef09d337d0ed5"}
May 26 16:01:26.178173 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:26.178147    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rg5xr" event={"ID":"0f99d273-701a-4186-836a-333aa040304b","Type":"ContainerStarted","Data":"0df4d75467b90c9f2d10524cb30f7aa62d28b9beaeab59fc40924778499aa9c6"}
May 26 16:01:26.180506 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:26.180481    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb" event={"ID":"26ce8382-d0aa-4fe6-8a61-7df596a1264f","Type":"ContainerStarted","Data":"1ef488103aa56fcce79c9847d338de2fd3c4b1e4f881509da599a7eb6d39fb7a"}
May 26 16:01:26.181368 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:26.181348    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t2rzl" event={"ID":"aaf93059-5a36-41c9-901e-fb7ae0e4224f","Type":"ContainerStarted","Data":"4d24632e13209455448c3767e6ade48523e6e6721ddf4275499fa9899f7e1ee5"}
May 26 16:01:26.182886 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:26.182868    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mz8tq" event={"ID":"ea5226b3-f441-403d-aa44-8ac33afd311b","Type":"ContainerStarted","Data":"d3080fdc7d78bed1cec7e52f85c4c40aec8df6e1eea86614eb9f2b6269c700f3"}
May 26 16:01:26.183850 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:26.183827    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nnztf" event={"ID":"1920789f-1573-4118-a9a4-7e3a44fcb1ad","Type":"ContainerStarted","Data":"839f18b62970b7bb3f3e41211cc86b5baebffb608e2dbce977beb39d23112df7"}
May 26 16:01:26.184850 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:26.184832    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp" event={"ID":"078ebf89-62fd-47e8-8a9f-03bf7431c882","Type":"ContainerStarted","Data":"3500e1bfa0fdc03348a12f1b65d0a04b9d4f55acad55c154202deb3bd6954f4f"}
May 26 16:01:26.186476 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:26.186454    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-193.ec2.internal" event={"ID":"e207af1d280d5e1f7a4ab2b460263c14","Type":"ContainerStarted","Data":"079d9800753d511c9a7373249f0f3e821db6a3d33c12fd5d48d7a746b8f39e77"}
May 26 16:01:26.187782 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:26.187761    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zgpnk" event={"ID":"fc50ed1b-7010-4636-b832-6fd7c64b57da","Type":"ContainerStarted","Data":"a0cfd2f70d3a3bd174d271809030bd88c70d5033525aa4852fda20b8dee09655"}
May 26 16:01:26.200620 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:26.200575    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-193.ec2.internal" podStartSLOduration=2.200564026 podStartE2EDuration="2.200564026s" podCreationTimestamp="2026-05-26 16:01:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:01:26.20027177 +0000 UTC m=+3.623864931" watchObservedRunningTime="2026-05-26 16:01:26.200564026 +0000 UTC m=+3.624157186"
May 26 16:01:26.644632 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:26.644582    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6d6\" (UniqueName: \"kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6\") pod \"network-check-target-dd7q4\" (UID: \"67ccb020-7afc-463c-a316-263722c743ad\") " pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:26.644808 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:26.644788    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:26.644808 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:26.644808    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:26.644918 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:26.644821    2560 projected.go:194] Error preparing data for projected volume kube-api-access-rk6d6 for pod openshift-network-diagnostics/network-check-target-dd7q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:26.644918 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:26.644879    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6 podName:67ccb020-7afc-463c-a316-263722c743ad nodeName:}" failed. No retries permitted until 2026-05-26 16:01:28.644860785 +0000 UTC m=+6.068453941 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rk6d6" (UniqueName: "kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6") pod "network-check-target-dd7q4" (UID: "67ccb020-7afc-463c-a316-263722c743ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:26.745782 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:26.745745    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs\") pod \"network-metrics-daemon-j528d\" (UID: \"22219f09-c007-46a5-b425-df729845bf45\") " pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:26.745947 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:26.745914    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:26.746009 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:26.745982    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs podName:22219f09-c007-46a5-b425-df729845bf45 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:28.74596169 +0000 UTC m=+6.169554852 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs") pod "network-metrics-daemon-j528d" (UID: "22219f09-c007-46a5-b425-df729845bf45") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:27.168795 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:27.168763    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:27.169242 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:27.168882    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:27.197821 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:27.197338    2560 generic.go:358] "Generic (PLEG): container finished" podID="a9b584bda4f897e7be9cad1c70cfd55b" containerID="5ffccc32c716c44cbdff86e0980994c9f2c77bc6a44b67a3b9b0f7e11bc64d7c" exitCode=0
May 26 16:01:27.197821 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:27.197546    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal" event={"ID":"a9b584bda4f897e7be9cad1c70cfd55b","Type":"ContainerDied","Data":"5ffccc32c716c44cbdff86e0980994c9f2c77bc6a44b67a3b9b0f7e11bc64d7c"}
May 26 16:01:28.166683 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:28.166645    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:28.166887 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:28.166790    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:28.202486 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:28.202368    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal" event={"ID":"a9b584bda4f897e7be9cad1c70cfd55b","Type":"ContainerStarted","Data":"20baf8d63a63a422dacfc06e4dfa39786b0cd433b851628051e6f03bd06a29d0"}
May 26 16:01:28.218547 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:28.218497    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-193.ec2.internal" podStartSLOduration=4.218481726 podStartE2EDuration="4.218481726s" podCreationTimestamp="2026-05-26 16:01:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:01:28.217679162 +0000 UTC m=+5.641272337" watchObservedRunningTime="2026-05-26 16:01:28.218481726 +0000 UTC m=+5.642074888"
May 26 16:01:28.662644 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:28.662602    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6d6\" (UniqueName: \"kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6\") pod \"network-check-target-dd7q4\" (UID: \"67ccb020-7afc-463c-a316-263722c743ad\") " pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:28.662823 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:28.662791    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:28.662823 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:28.662815    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:28.662941 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:28.662830    2560 projected.go:194] Error preparing data for projected volume kube-api-access-rk6d6 for pod openshift-network-diagnostics/network-check-target-dd7q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:28.662941 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:28.662889    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6 podName:67ccb020-7afc-463c-a316-263722c743ad nodeName:}" failed. No retries permitted until 2026-05-26 16:01:32.662869479 +0000 UTC m=+10.086462643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rk6d6" (UniqueName: "kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6") pod "network-check-target-dd7q4" (UID: "67ccb020-7afc-463c-a316-263722c743ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:28.763541 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:28.763501    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs\") pod \"network-metrics-daemon-j528d\" (UID: \"22219f09-c007-46a5-b425-df729845bf45\") " pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:28.763722 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:28.763655    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:28.763774 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:28.763724    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs podName:22219f09-c007-46a5-b425-df729845bf45 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:32.763710843 +0000 UTC m=+10.187303982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs") pod "network-metrics-daemon-j528d" (UID: "22219f09-c007-46a5-b425-df729845bf45") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:29.166312 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:29.166270    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:29.166485 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:29.166416    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:30.166604 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:30.166530    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:30.167069 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:30.166683    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:31.166884 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:31.166853    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:31.167238 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:31.166955    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:32.167067 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:32.167031    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:32.167519 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:32.167182    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:32.693836 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:32.693754    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6d6\" (UniqueName: \"kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6\") pod \"network-check-target-dd7q4\" (UID: \"67ccb020-7afc-463c-a316-263722c743ad\") " pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:32.693982 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:32.693934    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:32.693982 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:32.693960    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:32.693982 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:32.693973    2560 projected.go:194] Error preparing data for projected volume kube-api-access-rk6d6 for pod openshift-network-diagnostics/network-check-target-dd7q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:32.694114 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:32.694039    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6 podName:67ccb020-7afc-463c-a316-263722c743ad nodeName:}" failed. No retries permitted until 2026-05-26 16:01:40.694020424 +0000 UTC m=+18.117613571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rk6d6" (UniqueName: "kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6") pod "network-check-target-dd7q4" (UID: "67ccb020-7afc-463c-a316-263722c743ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:32.794748 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:32.794325    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs\") pod \"network-metrics-daemon-j528d\" (UID: \"22219f09-c007-46a5-b425-df729845bf45\") " pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:32.794748 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:32.794594    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:32.794748 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:32.794668    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs podName:22219f09-c007-46a5-b425-df729845bf45 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:40.794646084 +0000 UTC m=+18.218239246 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs") pod "network-metrics-daemon-j528d" (UID: "22219f09-c007-46a5-b425-df729845bf45") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:33.168523 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:33.167997    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:33.168523 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:33.168144    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:33.212333 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:33.212295    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xx9vq" event={"ID":"a1e36e32-7353-4703-8b16-848a49a47e91","Type":"ContainerStarted","Data":"b3ca8c342715be4bf4407c9ecc05baf4c1573dc8302c9b1180d3e34718158435"}
May 26 16:01:33.214933 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:33.214893    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb" event={"ID":"26ce8382-d0aa-4fe6-8a61-7df596a1264f","Type":"ContainerStarted","Data":"959ef04668339736b64b12ff99c69e15f40b9d43a085c87011c4f79da215be21"}
May 26 16:01:33.216932 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:33.216893    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t2rzl" event={"ID":"aaf93059-5a36-41c9-901e-fb7ae0e4224f","Type":"ContainerStarted","Data":"9140117df6410ea9de3f5565e179d274e5c4a71fb914b5100f8243e9996d92b1"}
May 26 16:01:33.218352 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:33.218151    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mz8tq" event={"ID":"ea5226b3-f441-403d-aa44-8ac33afd311b","Type":"ContainerStarted","Data":"7c8f32640baac979f03575ffe3decd47a56d278e4ce6fe3e8bb61cd706553ef6"}
May 26 16:01:33.228187 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:33.228063    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-xx9vq" podStartSLOduration=3.92214941 podStartE2EDuration="10.228049007s" podCreationTimestamp="2026-05-26 16:01:23 +0000 UTC" firstStartedPulling="2026-05-26 16:01:25.934004518 +0000 UTC m=+3.357597660" lastFinishedPulling="2026-05-26 16:01:32.239904104 +0000 UTC m=+9.663497257" observedRunningTime="2026-05-26 16:01:33.227294822 +0000 UTC m=+10.650887984" watchObservedRunningTime="2026-05-26 16:01:33.228049007 +0000 UTC m=+10.651642170"
May 26 16:01:33.241851 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:33.241782    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t2rzl" podStartSLOduration=3.985650998 podStartE2EDuration="10.241769418s" podCreationTimestamp="2026-05-26 16:01:23 +0000 UTC" firstStartedPulling="2026-05-26 16:01:25.938295614 +0000 UTC m=+3.361888768" lastFinishedPulling="2026-05-26 16:01:32.194414034 +0000 UTC m=+9.618007188" observedRunningTime="2026-05-26 16:01:33.241375633 +0000 UTC m=+10.664968796" watchObservedRunningTime="2026-05-26 16:01:33.241769418 +0000 UTC m=+10.665362581"
May 26 16:01:33.255353 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:33.255270    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mz8tq" podStartSLOduration=4.001309097 podStartE2EDuration="10.255255744s" podCreationTimestamp="2026-05-26 16:01:23 +0000 UTC" firstStartedPulling="2026-05-26 16:01:25.935008717 +0000 UTC m=+3.358601870" lastFinishedPulling="2026-05-26 16:01:32.188955358 +0000 UTC m=+9.612548517" observedRunningTime="2026-05-26 16:01:33.254766459 +0000 UTC m=+10.678359623" watchObservedRunningTime="2026-05-26 16:01:33.255255744 +0000 UTC m=+10.678848905"
May 26 16:01:34.166846 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:34.166698    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:34.166951 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:34.166928    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:34.187645 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:34.187621    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 26 16:01:34.221537 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:34.221444    2560 generic.go:358] "Generic (PLEG): container finished" podID="1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6" containerID="3f0a520d557843a4f76ffe46f824b7e80017597e917d494cf9a9a7c85e28ee90" exitCode=0
May 26 16:01:34.221652 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:34.221536    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vw726" event={"ID":"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6","Type":"ContainerDied","Data":"3f0a520d557843a4f76ffe46f824b7e80017597e917d494cf9a9a7c85e28ee90"}
May 26 16:01:34.224042 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:34.224022    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb" event={"ID":"26ce8382-d0aa-4fe6-8a61-7df596a1264f","Type":"ContainerStarted","Data":"0dd2042f0953eaf82db56a2246be4dbe1b0d48ffec336009a86aebf387d7c402"}
May 26 16:01:34.227243 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:34.227196    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nnztf" event={"ID":"1920789f-1573-4118-a9a4-7e3a44fcb1ad","Type":"ContainerStarted","Data":"0c2e16f18875b370c58a695a31e57bdea12aec1ae40545c0333700ca22b3b785"}
May 26 16:01:34.255903 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:34.255731    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nnztf" podStartSLOduration=4.93301461 podStartE2EDuration="11.255705187s" podCreationTimestamp="2026-05-26 16:01:23 +0000 UTC" firstStartedPulling="2026-05-26 16:01:25.928643123 +0000 UTC m=+3.352236262" lastFinishedPulling="2026-05-26 16:01:32.251333687 +0000 UTC m=+9.674926839" observedRunningTime="2026-05-26 16:01:34.255577856 +0000 UTC m=+11.679171021" watchObservedRunningTime="2026-05-26 16:01:34.255705187 +0000 UTC m=+11.679298382"
May 26 16:01:35.090313 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:35.090193    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-26T16:01:34.187640578Z","UUID":"489f27f8-9e32-4b78-b807-f2d9a557c9a8","Handler":null,"Name":"","Endpoint":""}
May 26 16:01:35.093022 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:35.092791    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 26 16:01:35.093022 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:35.092825    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 26 16:01:35.166950 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:35.166920    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:35.167102 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:35.167017    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:36.166240 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:36.166203    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:36.166854 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:36.166358    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:37.051798 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:37.051760    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t2rzl"
May 26 16:01:37.052434 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:37.052411    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t2rzl"
May 26 16:01:37.167108 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:37.167077    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:37.167763 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:37.167199    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:37.233320 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:37.233154    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t2rzl"
May 26 16:01:37.233703 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:37.233685    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t2rzl"
May 26 16:01:38.166614 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:38.166582    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:38.166791 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:38.166712    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:39.166787 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:39.166751    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:39.167311 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:39.166872    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:40.166369 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:40.166331    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:40.166606 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:40.166499    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:40.757864 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:40.757827    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6d6\" (UniqueName: \"kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6\") pod \"network-check-target-dd7q4\" (UID: \"67ccb020-7afc-463c-a316-263722c743ad\") " pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:40.758347 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:40.757977    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:40.758347 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:40.757995    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:40.758347 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:40.758004    2560 projected.go:194] Error preparing data for projected volume kube-api-access-rk6d6 for pod openshift-network-diagnostics/network-check-target-dd7q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:40.758347 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:40.758050    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6 podName:67ccb020-7afc-463c-a316-263722c743ad nodeName:}" failed. No retries permitted until 2026-05-26 16:01:56.758036395 +0000 UTC m=+34.181629534 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rk6d6" (UniqueName: "kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6") pod "network-check-target-dd7q4" (UID: "67ccb020-7afc-463c-a316-263722c743ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:40.858509 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:40.858473    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs\") pod \"network-metrics-daemon-j528d\" (UID: \"22219f09-c007-46a5-b425-df729845bf45\") " pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:40.858643 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:40.858586    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:40.858643 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:40.858634    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs podName:22219f09-c007-46a5-b425-df729845bf45 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:56.858621237 +0000 UTC m=+34.282214375 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs") pod "network-metrics-daemon-j528d" (UID: "22219f09-c007-46a5-b425-df729845bf45") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:41.166523 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:41.166461    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:41.166647 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:41.166556    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:42.166176 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:42.166138    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:42.166596 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:42.166276    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:43.171000 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:43.170679    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:43.171721 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:43.171683    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:44.167084 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:44.167052    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:44.167248 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:44.167187    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:45.166179 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:45.166145    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:45.166721 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:45.166261    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:46.166660 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:46.166630    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:46.167086 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:46.166772    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:47.166200 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:47.166162    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:47.166383 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:47.166279    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:48.166804 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:48.166778    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:48.167145 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:48.166934    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:48.261506 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:48.261287    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zgpnk" event={"ID":"fc50ed1b-7010-4636-b832-6fd7c64b57da","Type":"ContainerStarted","Data":"6401b79f2427605ccf7f85e2345a234a25e63c91c1b6e06bb99e496b9240cc86"}
May 26 16:01:48.262930 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:48.262905    2560 generic.go:358] "Generic (PLEG): container finished" podID="1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6" containerID="75eadf03e4751e9fe0b342716b0f816e5b2f39ea59a5e0e97764474be9c0c825" exitCode=0
May 26 16:01:48.263017 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:48.262983    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vw726" event={"ID":"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6","Type":"ContainerDied","Data":"75eadf03e4751e9fe0b342716b0f816e5b2f39ea59a5e0e97764474be9c0c825"}
May 26 16:01:48.264272 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:48.264191    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rg5xr" event={"ID":"0f99d273-701a-4186-836a-333aa040304b","Type":"ContainerStarted","Data":"7f38adfd83602eede3bade03a96cba38757d96eb7b14a359c6de56ce2c13b0e8"}
May 26 16:01:48.266021 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:48.265995    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb" event={"ID":"26ce8382-d0aa-4fe6-8a61-7df596a1264f","Type":"ContainerStarted","Data":"f0705f261a987b9080dcb7830ff0b7a4cb3473d5ec71b08e4b86b38f6dde3bf0"}
May 26 16:01:48.267678 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:48.267660    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp" event={"ID":"078ebf89-62fd-47e8-8a9f-03bf7431c882","Type":"ContainerStarted","Data":"a3026aad75713746d44540f9c4ba79427419f3633b4d5829333718a1c6600abd"}
May 26 16:01:48.267678 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:48.267682    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp" event={"ID":"078ebf89-62fd-47e8-8a9f-03bf7431c882","Type":"ContainerStarted","Data":"90aa8bfb3cad0deb60f45d1037437dbd3d72c920f5f801ca42efe700fc0433f9"}
May 26 16:01:48.267678 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:48.267691    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp" event={"ID":"078ebf89-62fd-47e8-8a9f-03bf7431c882","Type":"ContainerStarted","Data":"18776f0cb6966d54b48e93ed59dbb2a8975f57e35b7ebd6f068a3cb9fe64b482"}
May 26 16:01:48.276374 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:48.276344    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zgpnk" podStartSLOduration=3.345514933 podStartE2EDuration="25.276335387s" podCreationTimestamp="2026-05-26 16:01:23 +0000 UTC" firstStartedPulling="2026-05-26 16:01:25.936849708 +0000 UTC m=+3.360442861" lastFinishedPulling="2026-05-26 16:01:47.867670161 +0000 UTC m=+25.291263315" observedRunningTime="2026-05-26 16:01:48.276313778 +0000 UTC m=+25.699906938" watchObservedRunningTime="2026-05-26 16:01:48.276335387 +0000 UTC m=+25.699928547"
May 26 16:01:48.292009 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:48.291976    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-rg5xr" podStartSLOduration=3.351205382 podStartE2EDuration="25.291966541s" podCreationTimestamp="2026-05-26 16:01:23 +0000 UTC" firstStartedPulling="2026-05-26 16:01:25.926575969 +0000 UTC m=+3.350169108" lastFinishedPulling="2026-05-26 16:01:47.867337114 +0000 UTC m=+25.290930267" observedRunningTime="2026-05-26 16:01:48.291569032 +0000 UTC m=+25.715162194" watchObservedRunningTime="2026-05-26 16:01:48.291966541 +0000 UTC m=+25.715559701"
May 26 16:01:48.329814 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:48.329783    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxgzb" podStartSLOduration=3.3870963 podStartE2EDuration="25.329771673s" podCreationTimestamp="2026-05-26 16:01:23 +0000 UTC" firstStartedPulling="2026-05-26 16:01:25.927878052 +0000 UTC m=+3.351471197" lastFinishedPulling="2026-05-26 16:01:47.870553431 +0000 UTC m=+25.294146570" observedRunningTime="2026-05-26 16:01:48.329491813 +0000 UTC m=+25.753084974" watchObservedRunningTime="2026-05-26 16:01:48.329771673 +0000 UTC m=+25.753364834"
May 26 16:01:49.166953 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:49.166883    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:49.167525 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:49.166993    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:49.272105 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:49.272070    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp" event={"ID":"078ebf89-62fd-47e8-8a9f-03bf7431c882","Type":"ContainerStarted","Data":"a313f03c5f7903ccc40782ba570084e95297f6f719baad5c6685175d8689bb3c"}
May 26 16:01:49.272105 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:49.272107    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp" event={"ID":"078ebf89-62fd-47e8-8a9f-03bf7431c882","Type":"ContainerStarted","Data":"a59bd3dac0f2da370113c5dd676e0cc3c2e042d0b3404aab5909652184e3d322"}
May 26 16:01:49.272301 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:49.272120    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp" event={"ID":"078ebf89-62fd-47e8-8a9f-03bf7431c882","Type":"ContainerStarted","Data":"b4006b6930b81ceebacfa9debf530f1967aca2573649876b3b915b45a817c00d"}
May 26 16:01:50.166894 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:50.166864    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:50.167117 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:50.166974    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:50.275743 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:50.275709    2560 generic.go:358] "Generic (PLEG): container finished" podID="1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6" containerID="eccee1247ad2e6b8dd49af48a57519f2b37045ae4394def353f4df2b2c655f37" exitCode=0
May 26 16:01:50.275918 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:50.275763    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vw726" event={"ID":"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6","Type":"ContainerDied","Data":"eccee1247ad2e6b8dd49af48a57519f2b37045ae4394def353f4df2b2c655f37"}
May 26 16:01:51.166982 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:51.166948    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:51.167154 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:51.167067    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:51.279690 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:51.279659    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp" event={"ID":"078ebf89-62fd-47e8-8a9f-03bf7431c882","Type":"ContainerStarted","Data":"82e0a699130259ced0532b5080e78173d2d21073ce1f52ae6f479dfe2cb6686d"}
May 26 16:01:52.166576 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:52.166544    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:52.166723 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:52.166651    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:52.284161 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:52.284130    2560 generic.go:358] "Generic (PLEG): container finished" podID="1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6" containerID="e1b73fc51fc7f3c6dbf73c2fd067bbc111768d377cfdfd2453f2cf0d873f4c64" exitCode=0
May 26 16:01:52.284572 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:52.284200    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vw726" event={"ID":"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6","Type":"ContainerDied","Data":"e1b73fc51fc7f3c6dbf73c2fd067bbc111768d377cfdfd2453f2cf0d873f4c64"}
May 26 16:01:53.167570 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:53.167292    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:53.167741 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:53.167590    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:53.289869 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:53.289829    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp" event={"ID":"078ebf89-62fd-47e8-8a9f-03bf7431c882","Type":"ContainerStarted","Data":"6202fb2fe56fde97205a14a60fbd2f7d894510ea62c29f03485dec26d87fa81c"}
May 26 16:01:53.290446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:53.290186    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:53.309136 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:53.309112    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:53.316794 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:53.316757    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp" podStartSLOduration=8.359816132 podStartE2EDuration="30.316744234s" podCreationTimestamp="2026-05-26 16:01:23 +0000 UTC" firstStartedPulling="2026-05-26 16:01:25.925038613 +0000 UTC m=+3.348631766" lastFinishedPulling="2026-05-26 16:01:47.881966715 +0000 UTC m=+25.305559868" observedRunningTime="2026-05-26 16:01:53.316598584 +0000 UTC m=+30.740191749" watchObservedRunningTime="2026-05-26 16:01:53.316744234 +0000 UTC m=+30.740337396"
May 26 16:01:54.166485 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:54.166455    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:54.166646 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:54.166607    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:54.292207 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:54.292172    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:54.292207 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:54.292215    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:54.309533 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:54.309506    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:01:54.788145 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:54.788064    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j528d"]
May 26 16:01:54.788301 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:54.788175    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:54.788360 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:54.788298    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:54.799781 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:54.799756    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dd7q4"]
May 26 16:01:54.799894 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:54.799870    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:54.799982 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:54.799963    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:56.166700 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:56.166666    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:56.167133 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:56.166789    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:56.766646 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:56.766600    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6d6\" (UniqueName: \"kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6\") pod \"network-check-target-dd7q4\" (UID: \"67ccb020-7afc-463c-a316-263722c743ad\") " pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:56.766837 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:56.766814    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:56.766904 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:56.766845    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:56.766904 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:56.766860    2560 projected.go:194] Error preparing data for projected volume kube-api-access-rk6d6 for pod openshift-network-diagnostics/network-check-target-dd7q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:56.766965 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:56.766927    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6 podName:67ccb020-7afc-463c-a316-263722c743ad nodeName:}" failed. No retries permitted until 2026-05-26 16:02:28.766908293 +0000 UTC m=+66.190501436 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rk6d6" (UniqueName: "kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6") pod "network-check-target-dd7q4" (UID: "67ccb020-7afc-463c-a316-263722c743ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:56.867613 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:56.867569    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs\") pod \"network-metrics-daemon-j528d\" (UID: \"22219f09-c007-46a5-b425-df729845bf45\") " pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:56.867769 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:56.867744    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:56.867831 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:56.867815    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs podName:22219f09-c007-46a5-b425-df729845bf45 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:28.867793233 +0000 UTC m=+66.291386372 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs") pod "network-metrics-daemon-j528d" (UID: "22219f09-c007-46a5-b425-df729845bf45") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:57.166342 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:57.166235    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:57.166527 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:57.166405    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:58.166539 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:58.166506    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:01:58.166874 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:58.166601    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:01:59.166487 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:59.166244    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:01:59.166630 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:01:59.166525    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:01:59.302742 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:59.302706    2560 generic.go:358] "Generic (PLEG): container finished" podID="1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6" containerID="ab2ba077b60a24413968ad1e34e1dad77dec4b15b742b6fde7b7a0c4d6ea1cf6" exitCode=0
May 26 16:01:59.302742 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:01:59.302744    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vw726" event={"ID":"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6","Type":"ContainerDied","Data":"ab2ba077b60a24413968ad1e34e1dad77dec4b15b742b6fde7b7a0c4d6ea1cf6"}
May 26 16:02:00.166189 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.166161    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:02:00.166333 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:00.166262    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd7q4" podUID="67ccb020-7afc-463c-a316-263722c743ad"
May 26 16:02:00.307502 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.307469    2560 generic.go:358] "Generic (PLEG): container finished" podID="1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6" containerID="f419c5310622aaa102c9a6cedaafc9be908ffcb02fa029b8f278f13add01c3ce" exitCode=0
May 26 16:02:00.307910 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.307525    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vw726" event={"ID":"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6","Type":"ContainerDied","Data":"f419c5310622aaa102c9a6cedaafc9be908ffcb02fa029b8f278f13add01c3ce"}
May 26 16:02:00.430078 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.430021    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-193.ec2.internal" event="NodeReady"
May 26 16:02:00.430195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.430168    2560 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 26 16:02:00.478366 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.478339    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v7sq4"]
May 26 16:02:00.509977 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.509901    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-29s87"]
May 26 16:02:00.510063 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.510026    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:00.512329 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.512304    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 26 16:02:00.512329 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.512317    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 26 16:02:00.512462 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.512367    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lppkw\""
May 26 16:02:00.529756 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.529735    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v7sq4"]
May 26 16:02:00.529756 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.529756    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-29s87"]
May 26 16:02:00.529865 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.529833    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:02:00.531818 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.531803    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 26 16:02:00.531945 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.531893    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 26 16:02:00.532009 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.531947    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 26 16:02:00.532009 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.531974    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-22s8h\""
May 26 16:02:00.596421 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.596385    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:00.596513 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.596497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wtr8\" (UniqueName: \"kubernetes.io/projected/b7627daa-7d0e-4a02-9b00-a3515e6b7137-kube-api-access-5wtr8\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:00.596566 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.596527    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7627daa-7d0e-4a02-9b00-a3515e6b7137-tmp-dir\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:00.596566 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.596552    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b7627daa-7d0e-4a02-9b00-a3515e6b7137-config-volume\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:00.697635 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.697578    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5wtr8\" (UniqueName: \"kubernetes.io/projected/b7627daa-7d0e-4a02-9b00-a3515e6b7137-kube-api-access-5wtr8\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:00.697635 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.697607    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7627daa-7d0e-4a02-9b00-a3515e6b7137-tmp-dir\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:00.697635 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.697628    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b7627daa-7d0e-4a02-9b00-a3515e6b7137-config-volume\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:00.697898 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.697879    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:00.697945 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.697917    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert\") pod \"ingress-canary-29s87\" (UID: \"fcbf8b71-6b16-4832-b58b-537691709f83\") " pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:02:00.697979 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:00.697944    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:02:00.697979 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.697951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qph7b\" (UniqueName: \"kubernetes.io/projected/fcbf8b71-6b16-4832-b58b-537691709f83-kube-api-access-qph7b\") pod \"ingress-canary-29s87\" (UID: \"fcbf8b71-6b16-4832-b58b-537691709f83\") " pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:02:00.698038 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.697980    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7627daa-7d0e-4a02-9b00-a3515e6b7137-tmp-dir\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:00.698038 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:00.698001    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls podName:b7627daa-7d0e-4a02-9b00-a3515e6b7137 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:01.1979839 +0000 UTC m=+38.621577039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls") pod "dns-default-v7sq4" (UID: "b7627daa-7d0e-4a02-9b00-a3515e6b7137") : secret "dns-default-metrics-tls" not found
May 26 16:02:00.698188 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.698173    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b7627daa-7d0e-4a02-9b00-a3515e6b7137-config-volume\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:00.710213 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.710186    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wtr8\" (UniqueName: \"kubernetes.io/projected/b7627daa-7d0e-4a02-9b00-a3515e6b7137-kube-api-access-5wtr8\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:00.798259 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.798236    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert\") pod \"ingress-canary-29s87\" (UID: \"fcbf8b71-6b16-4832-b58b-537691709f83\") " pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:02:00.798355 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.798265    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qph7b\" (UniqueName: \"kubernetes.io/projected/fcbf8b71-6b16-4832-b58b-537691709f83-kube-api-access-qph7b\") pod \"ingress-canary-29s87\" (UID: \"fcbf8b71-6b16-4832-b58b-537691709f83\") " pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:02:00.798414 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:00.798381    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:02:00.798485 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:00.798474    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert podName:fcbf8b71-6b16-4832-b58b-537691709f83 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:01.298455516 +0000 UTC m=+38.722048661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert") pod "ingress-canary-29s87" (UID: "fcbf8b71-6b16-4832-b58b-537691709f83") : secret "canary-serving-cert" not found
May 26 16:02:00.807001 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:00.806979    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qph7b\" (UniqueName: \"kubernetes.io/projected/fcbf8b71-6b16-4832-b58b-537691709f83-kube-api-access-qph7b\") pod \"ingress-canary-29s87\" (UID: \"fcbf8b71-6b16-4832-b58b-537691709f83\") " pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:02:01.166944 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:01.166913    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:02:01.169428 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:01.169404    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8ddfl\""
May 26 16:02:01.169570 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:01.169382    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 16:02:01.202196 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:01.202174    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:01.202357 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:01.202314    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:02:01.202435 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:01.202381    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls podName:b7627daa-7d0e-4a02-9b00-a3515e6b7137 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:02.202365623 +0000 UTC m=+39.625958761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls") pod "dns-default-v7sq4" (UID: "b7627daa-7d0e-4a02-9b00-a3515e6b7137") : secret "dns-default-metrics-tls" not found
May 26 16:02:01.303296 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:01.303261    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert\") pod \"ingress-canary-29s87\" (UID: \"fcbf8b71-6b16-4832-b58b-537691709f83\") " pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:02:01.303479 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:01.303372    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:02:01.303479 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:01.303472    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert podName:fcbf8b71-6b16-4832-b58b-537691709f83 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:02.303454167 +0000 UTC m=+39.727047310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert") pod "ingress-canary-29s87" (UID: "fcbf8b71-6b16-4832-b58b-537691709f83") : secret "canary-serving-cert" not found
May 26 16:02:01.312068 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:01.312044    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vw726" event={"ID":"1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6","Type":"ContainerStarted","Data":"5a10279ac689be59ffa1bd87f2d12fcb26120f95c4fa59b848856526e35948fd"}
May 26 16:02:01.332911 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:01.332870    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vw726" podStartSLOduration=5.69140217 podStartE2EDuration="38.332859972s" podCreationTimestamp="2026-05-26 16:01:23 +0000 UTC" firstStartedPulling="2026-05-26 16:01:25.932973041 +0000 UTC m=+3.356566192" lastFinishedPulling="2026-05-26 16:01:58.574430855 +0000 UTC m=+35.998023994" observedRunningTime="2026-05-26 16:02:01.331448084 +0000 UTC m=+38.755041246" watchObservedRunningTime="2026-05-26 16:02:01.332859972 +0000 UTC m=+38.756453133"
May 26 16:02:02.166672 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:02.166633    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:02:02.169731 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:02.169705    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 16:02:02.169731 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:02.169706    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 16:02:02.169972 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:02.169706    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45wxn\""
May 26 16:02:02.209738 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:02.209701    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:02.209883 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:02.209835    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:02:02.209933 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:02.209896    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls podName:b7627daa-7d0e-4a02-9b00-a3515e6b7137 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:04.209880545 +0000 UTC m=+41.633473684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls") pod "dns-default-v7sq4" (UID: "b7627daa-7d0e-4a02-9b00-a3515e6b7137") : secret "dns-default-metrics-tls" not found
May 26 16:02:02.310379 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:02.310338    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert\") pod \"ingress-canary-29s87\" (UID: \"fcbf8b71-6b16-4832-b58b-537691709f83\") " pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:02:02.310548 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:02.310495    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:02:02.310586 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:02.310560    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert podName:fcbf8b71-6b16-4832-b58b-537691709f83 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:04.310543764 +0000 UTC m=+41.734136904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert") pod "ingress-canary-29s87" (UID: "fcbf8b71-6b16-4832-b58b-537691709f83") : secret "canary-serving-cert" not found
May 26 16:02:04.227440 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:04.227381    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:04.227953 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:04.227542    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:02:04.227953 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:04.227609    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls podName:b7627daa-7d0e-4a02-9b00-a3515e6b7137 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:08.227590588 +0000 UTC m=+45.651183727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls") pod "dns-default-v7sq4" (UID: "b7627daa-7d0e-4a02-9b00-a3515e6b7137") : secret "dns-default-metrics-tls" not found
May 26 16:02:04.328506 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:04.328482    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert\") pod \"ingress-canary-29s87\" (UID: \"fcbf8b71-6b16-4832-b58b-537691709f83\") " pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:02:04.328655 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:04.328636    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:02:04.328723 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:04.328712    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert podName:fcbf8b71-6b16-4832-b58b-537691709f83 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:08.328691295 +0000 UTC m=+45.752284433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert") pod "ingress-canary-29s87" (UID: "fcbf8b71-6b16-4832-b58b-537691709f83") : secret "canary-serving-cert" not found
May 26 16:02:08.254796 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:08.254756    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:08.255333 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:08.254931    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:02:08.255333 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:08.255010    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls podName:b7627daa-7d0e-4a02-9b00-a3515e6b7137 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:16.254989321 +0000 UTC m=+53.678582466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls") pod "dns-default-v7sq4" (UID: "b7627daa-7d0e-4a02-9b00-a3515e6b7137") : secret "dns-default-metrics-tls" not found
May 26 16:02:08.355851 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:08.355822    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert\") pod \"ingress-canary-29s87\" (UID: \"fcbf8b71-6b16-4832-b58b-537691709f83\") " pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:02:08.355983 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:08.355963    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:02:08.356030 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:08.356019    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert podName:fcbf8b71-6b16-4832-b58b-537691709f83 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:16.356005405 +0000 UTC m=+53.779598544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert") pod "ingress-canary-29s87" (UID: "fcbf8b71-6b16-4832-b58b-537691709f83") : secret "canary-serving-cert" not found
May 26 16:02:16.314614 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:16.314571    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:16.314985 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:16.314714    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:02:16.314985 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:16.314780    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls podName:b7627daa-7d0e-4a02-9b00-a3515e6b7137 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:32.314763919 +0000 UTC m=+69.738357062 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls") pod "dns-default-v7sq4" (UID: "b7627daa-7d0e-4a02-9b00-a3515e6b7137") : secret "dns-default-metrics-tls" not found
May 26 16:02:16.415077 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:16.415045    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert\") pod \"ingress-canary-29s87\" (UID: \"fcbf8b71-6b16-4832-b58b-537691709f83\") " pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:02:16.415339 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:16.415210    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:02:16.415339 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:16.415283    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert podName:fcbf8b71-6b16-4832-b58b-537691709f83 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:32.415263417 +0000 UTC m=+69.838856575 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert") pod "ingress-canary-29s87" (UID: "fcbf8b71-6b16-4832-b58b-537691709f83") : secret "canary-serving-cert" not found
May 26 16:02:26.305357 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:26.305327    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n84fp"
May 26 16:02:28.797033 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:28.796998    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6d6\" (UniqueName: \"kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6\") pod \"network-check-target-dd7q4\" (UID: \"67ccb020-7afc-463c-a316-263722c743ad\") " pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:02:28.799637 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:28.799611    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 16:02:28.810257 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:28.810244    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 16:02:28.821434 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:28.821416    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6d6\" (UniqueName: \"kubernetes.io/projected/67ccb020-7afc-463c-a316-263722c743ad-kube-api-access-rk6d6\") pod \"network-check-target-dd7q4\" (UID: \"67ccb020-7afc-463c-a316-263722c743ad\") " pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:02:28.876796 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:28.876767    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45wxn\""
May 26 16:02:28.885510 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:28.885494    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:02:28.897592 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:28.897562    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs\") pod \"network-metrics-daemon-j528d\" (UID: \"22219f09-c007-46a5-b425-df729845bf45\") " pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:02:28.900254 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:28.900236    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 16:02:28.908010 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:28.907988    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 16:02:28.908113 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:28.908102    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs podName:22219f09-c007-46a5-b425-df729845bf45 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:32.908082036 +0000 UTC m=+130.331675190 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs") pod "network-metrics-daemon-j528d" (UID: "22219f09-c007-46a5-b425-df729845bf45") : secret "metrics-daemon-secret" not found
May 26 16:02:29.070895 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:29.070826    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dd7q4"]
May 26 16:02:29.077452 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:02:29.076818    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ccb020_7afc_463c_a316_263722c743ad.slice/crio-df35f77fe5be3828d826168575fd81e4633d05049128503989bb01900714b7a5 WatchSource:0}: Error finding container df35f77fe5be3828d826168575fd81e4633d05049128503989bb01900714b7a5: Status 404 returned error can't find the container with id df35f77fe5be3828d826168575fd81e4633d05049128503989bb01900714b7a5
May 26 16:02:29.361020 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:29.360946    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dd7q4" event={"ID":"67ccb020-7afc-463c-a316-263722c743ad","Type":"ContainerStarted","Data":"df35f77fe5be3828d826168575fd81e4633d05049128503989bb01900714b7a5"}
May 26 16:02:32.323092 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:32.323003    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:02:32.323462 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:32.323147    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:02:32.323462 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:32.323219    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls podName:b7627daa-7d0e-4a02-9b00-a3515e6b7137 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:04.323202723 +0000 UTC m=+101.746795863 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls") pod "dns-default-v7sq4" (UID: "b7627daa-7d0e-4a02-9b00-a3515e6b7137") : secret "dns-default-metrics-tls" not found
May 26 16:02:32.370987 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:32.370958    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dd7q4" event={"ID":"67ccb020-7afc-463c-a316-263722c743ad","Type":"ContainerStarted","Data":"5e97ae7661534e3b67023d70569025e6406e8a94f8da1c6855bff1afdf07bb10"}
May 26 16:02:32.371122 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:32.371111    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:02:32.386195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:32.386156    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dd7q4" podStartSLOduration=66.425293933 podStartE2EDuration="1m9.3861443s" podCreationTimestamp="2026-05-26 16:01:23 +0000 UTC" firstStartedPulling="2026-05-26 16:02:29.078269524 +0000 UTC m=+66.501862665" lastFinishedPulling="2026-05-26 16:02:32.03911987 +0000 UTC m=+69.462713032" observedRunningTime="2026-05-26 16:02:32.385937257 +0000 UTC m=+69.809530420" watchObservedRunningTime="2026-05-26 16:02:32.3861443 +0000 UTC m=+69.809737455"
May 26 16:02:32.423626 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:02:32.423599    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert\") pod \"ingress-canary-29s87\" (UID: \"fcbf8b71-6b16-4832-b58b-537691709f83\") " pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:02:32.423732 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:32.423686    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:02:32.423776 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:02:32.423758    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert podName:fcbf8b71-6b16-4832-b58b-537691709f83 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:04.423743875 +0000 UTC m=+101.847337013 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert") pod "ingress-canary-29s87" (UID: "fcbf8b71-6b16-4832-b58b-537691709f83") : secret "canary-serving-cert" not found
May 26 16:03:03.375209 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:03.375101    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dd7q4"
May 26 16:03:04.343169 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:04.343139    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:03:04.343297 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:04.343282    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:03:04.343348 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:04.343339    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls podName:b7627daa-7d0e-4a02-9b00-a3515e6b7137 nodeName:}" failed. No retries permitted until 2026-05-26 16:04:08.343324373 +0000 UTC m=+165.766917518 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls") pod "dns-default-v7sq4" (UID: "b7627daa-7d0e-4a02-9b00-a3515e6b7137") : secret "dns-default-metrics-tls" not found
May 26 16:03:04.444512 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:04.444477    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert\") pod \"ingress-canary-29s87\" (UID: \"fcbf8b71-6b16-4832-b58b-537691709f83\") " pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:03:04.444866 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:04.444590    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:03:04.444866 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:04.444650    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert podName:fcbf8b71-6b16-4832-b58b-537691709f83 nodeName:}" failed. No retries permitted until 2026-05-26 16:04:08.444636997 +0000 UTC m=+165.868230136 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert") pod "ingress-canary-29s87" (UID: "fcbf8b71-6b16-4832-b58b-537691709f83") : secret "canary-serving-cert" not found
May 26 16:03:31.738603 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.738573    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-j57x7"]
May 26 16:03:31.740279 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.740259    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-j57x7"
May 26 16:03:31.742701 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.742679    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 26 16:03:31.742805 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.742700    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 26 16:03:31.743535 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.743522    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6mdb7\""
May 26 16:03:31.751536 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.751503    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-j57x7"]
May 26 16:03:31.821065 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.821026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f74fb\" (UniqueName: \"kubernetes.io/projected/7d3526bf-6f30-4cd2-b6a9-44a0dfc32777-kube-api-access-f74fb\") pod \"volume-data-source-validator-5cf8c89d64-j57x7\" (UID: \"7d3526bf-6f30-4cd2-b6a9-44a0dfc32777\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-j57x7"
May 26 16:03:31.821782 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.821756    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t"]
May 26 16:03:31.823715 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.823700    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-vkhrc"]
May 26 16:03:31.823868 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.823850    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t"
May 26 16:03:31.825996 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.825976    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"]
May 26 16:03:31.826135 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.826120    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:31.826280 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.826262    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
May 26 16:03:31.826364 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.826293    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
May 26 16:03:31.826609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.826586    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
May 26 16:03:31.826751 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.826618    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
May 26 16:03:31.826751 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.826631    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-dkwrh\""
May 26 16:03:31.827954 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.827933    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:03:31.828584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.828568    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 26 16:03:31.828584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.828573    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tmbsg\""
May 26 16:03:31.828856 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.828843    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 26 16:03:31.828902 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.828860    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 26 16:03:31.829520 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.829501    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 26 16:03:31.830606 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.830404    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
May 26 16:03:31.830606 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.830406    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 26 16:03:31.831632 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.831616    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5zf4h\""
May 26 16:03:31.831732 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.831705    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 26 16:03:31.831798 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.831783    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
May 26 16:03:31.835725 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.835708    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 26 16:03:31.841944 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.841926    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t"]
May 26 16:03:31.843050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.843030    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-vkhrc"]
May 26 16:03:31.872484 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.872454    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"]
May 26 16:03:31.922094 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.922066    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wrwg\" (UniqueName: \"kubernetes.io/projected/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-kube-api-access-9wrwg\") pod \"cluster-monitoring-operator-6c75959455-9zl7z\" (UID: \"7028a76e-6db4-4b2c-afd5-d7ab066dc10a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:03:31.922479 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.922456    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8ad1242b-3f03-48c6-a0cb-f029836842ff-config\") pod \"console-operator-76dff48d59-vkhrc\" (UID: \"8ad1242b-3f03-48c6-a0cb-f029836842ff\") " pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:31.922737 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.922718    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-442jh\" (UniqueName: \"kubernetes.io/projected/8ad1242b-3f03-48c6-a0cb-f029836842ff-kube-api-access-442jh\") pod \"console-operator-76dff48d59-vkhrc\" (UID: \"8ad1242b-3f03-48c6-a0cb-f029836842ff\") " pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:31.922954 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.922940    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f74fb\" (UniqueName: \"kubernetes.io/projected/7d3526bf-6f30-4cd2-b6a9-44a0dfc32777-kube-api-access-f74fb\") pod \"volume-data-source-validator-5cf8c89d64-j57x7\" (UID: \"7d3526bf-6f30-4cd2-b6a9-44a0dfc32777\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-j57x7"
May 26 16:03:31.923464 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.923444    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-9zl7z\" (UID: \"7028a76e-6db4-4b2c-afd5-d7ab066dc10a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:03:31.923687 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.923670    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4f1c38-12f6-4b6c-9360-0ba547d5f28a-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-mmd9t\" (UID: \"8f4f1c38-12f6-4b6c-9360-0ba547d5f28a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t"
May 26 16:03:31.923929 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.923906    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4897d\" (UniqueName: \"kubernetes.io/projected/8f4f1c38-12f6-4b6c-9360-0ba547d5f28a-kube-api-access-4897d\") pod \"kube-storage-version-migrator-operator-565644f49-mmd9t\" (UID: \"8f4f1c38-12f6-4b6c-9360-0ba547d5f28a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t"
May 26 16:03:31.924198 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.924184    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ad1242b-3f03-48c6-a0cb-f029836842ff-serving-cert\") pod \"console-operator-76dff48d59-vkhrc\" (UID: \"8ad1242b-3f03-48c6-a0cb-f029836842ff\") " pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:31.924345 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.924331    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-9zl7z\" (UID: \"7028a76e-6db4-4b2c-afd5-d7ab066dc10a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:03:31.924558 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.924545    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f4f1c38-12f6-4b6c-9360-0ba547d5f28a-config\") pod \"kube-storage-version-migrator-operator-565644f49-mmd9t\" (UID: \"8f4f1c38-12f6-4b6c-9360-0ba547d5f28a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t"
May 26 16:03:31.924753 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.924738    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ad1242b-3f03-48c6-a0cb-f029836842ff-trusted-ca\") pod \"console-operator-76dff48d59-vkhrc\" (UID: \"8ad1242b-3f03-48c6-a0cb-f029836842ff\") " pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:31.925734 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.925718    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86594ddd4-x9sdj"]
May 26 16:03:31.929584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.929550    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:31.932700 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.932680    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 26 16:03:31.933030 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.933014    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 26 16:03:31.933371 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.933355    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 26 16:03:31.933729 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.933713    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nv2cx\""
May 26 16:03:31.944071 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.944042    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86594ddd4-x9sdj"]
May 26 16:03:31.945083 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.945065    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 26 16:03:31.946573 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:31.946552    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f74fb\" (UniqueName: \"kubernetes.io/projected/7d3526bf-6f30-4cd2-b6a9-44a0dfc32777-kube-api-access-f74fb\") pod \"volume-data-source-validator-5cf8c89d64-j57x7\" (UID: \"7d3526bf-6f30-4cd2-b6a9-44a0dfc32777\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-j57x7"
May 26 16:03:32.025565 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.025489    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9wrwg\" (UniqueName: \"kubernetes.io/projected/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-kube-api-access-9wrwg\") pod \"cluster-monitoring-operator-6c75959455-9zl7z\" (UID: \"7028a76e-6db4-4b2c-afd5-d7ab066dc10a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:03:32.025565 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.025531    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19de7b34-20ec-4840-8901-96b1c50c1964-ca-trust-extracted\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.025738 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.025567    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19de7b34-20ec-4840-8901-96b1c50c1964-registry-certificates\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.025738 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.025607    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8ad1242b-3f03-48c6-a0cb-f029836842ff-config\") pod \"console-operator-76dff48d59-vkhrc\" (UID: \"8ad1242b-3f03-48c6-a0cb-f029836842ff\") " pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:32.025738 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.025632    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19de7b34-20ec-4840-8901-96b1c50c1964-installation-pull-secrets\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.025738 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.025657    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gk92\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-kube-api-access-9gk92\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.025738 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.025716    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-442jh\" (UniqueName: \"kubernetes.io/projected/8ad1242b-3f03-48c6-a0cb-f029836842ff-kube-api-access-442jh\") pod \"console-operator-76dff48d59-vkhrc\" (UID: \"8ad1242b-3f03-48c6-a0cb-f029836842ff\") " pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:32.025887 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.025800    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19de7b34-20ec-4840-8901-96b1c50c1964-image-registry-private-configuration\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.025887 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.025841    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19de7b34-20ec-4840-8901-96b1c50c1964-trusted-ca\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.025887 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.025859    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.025887 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.025878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-bound-sa-token\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.026005 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.025914    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-9zl7z\" (UID: \"7028a76e-6db4-4b2c-afd5-d7ab066dc10a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:03:32.026005 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.025954    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4f1c38-12f6-4b6c-9360-0ba547d5f28a-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-mmd9t\" (UID: \"8f4f1c38-12f6-4b6c-9360-0ba547d5f28a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t"
May 26 16:03:32.026074 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:32.026023    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 16:03:32.026111 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:32.026078    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls podName:7028a76e-6db4-4b2c-afd5-d7ab066dc10a nodeName:}" failed. No retries permitted until 2026-05-26 16:03:32.526064557 +0000 UTC m=+129.949657696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-9zl7z" (UID: "7028a76e-6db4-4b2c-afd5-d7ab066dc10a") : secret "cluster-monitoring-operator-tls" not found
May 26 16:03:32.026163 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.026128    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4897d\" (UniqueName: \"kubernetes.io/projected/8f4f1c38-12f6-4b6c-9360-0ba547d5f28a-kube-api-access-4897d\") pod \"kube-storage-version-migrator-operator-565644f49-mmd9t\" (UID: \"8f4f1c38-12f6-4b6c-9360-0ba547d5f28a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t"
May 26 16:03:32.026208 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.026182    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ad1242b-3f03-48c6-a0cb-f029836842ff-serving-cert\") pod \"console-operator-76dff48d59-vkhrc\" (UID: \"8ad1242b-3f03-48c6-a0cb-f029836842ff\") " pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:32.026260 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.026209    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-9zl7z\" (UID: \"7028a76e-6db4-4b2c-afd5-d7ab066dc10a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:03:32.026260 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.026228    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8ad1242b-3f03-48c6-a0cb-f029836842ff-config\") pod \"console-operator-76dff48d59-vkhrc\" (UID: \"8ad1242b-3f03-48c6-a0cb-f029836842ff\") " pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:32.026260 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.026245    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f4f1c38-12f6-4b6c-9360-0ba547d5f28a-config\") pod \"kube-storage-version-migrator-operator-565644f49-mmd9t\" (UID: \"8f4f1c38-12f6-4b6c-9360-0ba547d5f28a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t"
May 26 16:03:32.026437 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.026270    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ad1242b-3f03-48c6-a0cb-f029836842ff-trusted-ca\") pod \"console-operator-76dff48d59-vkhrc\" (UID: \"8ad1242b-3f03-48c6-a0cb-f029836842ff\") " pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:32.027004 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.026980    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f4f1c38-12f6-4b6c-9360-0ba547d5f28a-config\") pod \"kube-storage-version-migrator-operator-565644f49-mmd9t\" (UID: \"8f4f1c38-12f6-4b6c-9360-0ba547d5f28a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t"
May 26 16:03:32.027456 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.027431    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-9zl7z\" (UID: \"7028a76e-6db4-4b2c-afd5-d7ab066dc10a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:03:32.028059 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.028037    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ad1242b-3f03-48c6-a0cb-f029836842ff-trusted-ca\") pod \"console-operator-76dff48d59-vkhrc\" (UID: \"8ad1242b-3f03-48c6-a0cb-f029836842ff\") " pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:32.029050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.029027    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4f1c38-12f6-4b6c-9360-0ba547d5f28a-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-mmd9t\" (UID: \"8f4f1c38-12f6-4b6c-9360-0ba547d5f28a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t"
May 26 16:03:32.029170 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.029150    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ad1242b-3f03-48c6-a0cb-f029836842ff-serving-cert\") pod \"console-operator-76dff48d59-vkhrc\" (UID: \"8ad1242b-3f03-48c6-a0cb-f029836842ff\") " pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:32.036756 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.036730    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4897d\" (UniqueName: \"kubernetes.io/projected/8f4f1c38-12f6-4b6c-9360-0ba547d5f28a-kube-api-access-4897d\") pod \"kube-storage-version-migrator-operator-565644f49-mmd9t\" (UID: \"8f4f1c38-12f6-4b6c-9360-0ba547d5f28a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t"
May 26 16:03:32.036826 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.036734    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wrwg\" (UniqueName: \"kubernetes.io/projected/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-kube-api-access-9wrwg\") pod \"cluster-monitoring-operator-6c75959455-9zl7z\" (UID: \"7028a76e-6db4-4b2c-afd5-d7ab066dc10a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:03:32.036924 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.036907    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-442jh\" (UniqueName: \"kubernetes.io/projected/8ad1242b-3f03-48c6-a0cb-f029836842ff-kube-api-access-442jh\") pod \"console-operator-76dff48d59-vkhrc\" (UID: \"8ad1242b-3f03-48c6-a0cb-f029836842ff\") " pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:32.049766 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.049751    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-j57x7"
May 26 16:03:32.126894 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.126864    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19de7b34-20ec-4840-8901-96b1c50c1964-ca-trust-extracted\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.127172 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.126910    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19de7b34-20ec-4840-8901-96b1c50c1964-registry-certificates\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.127172 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.127048    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19de7b34-20ec-4840-8901-96b1c50c1964-installation-pull-secrets\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.127172 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.127091    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9gk92\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-kube-api-access-9gk92\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.127172 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.127130    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19de7b34-20ec-4840-8901-96b1c50c1964-image-registry-private-configuration\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.127408 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.127173    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19de7b34-20ec-4840-8901-96b1c50c1964-trusted-ca\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.127408 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.127201    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.127408 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.127225    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-bound-sa-token\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.127408 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.127303    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19de7b34-20ec-4840-8901-96b1c50c1964-ca-trust-extracted\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.127613 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.127591    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19de7b34-20ec-4840-8901-96b1c50c1964-registry-certificates\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.127657 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:32.127606    2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:03:32.127657 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:32.127624    2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86594ddd4-x9sdj: secret "image-registry-tls" not found
May 26 16:03:32.127717 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:32.127678    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls podName:19de7b34-20ec-4840-8901-96b1c50c1964 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:32.62766373 +0000 UTC m=+130.051256869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls") pod "image-registry-86594ddd4-x9sdj" (UID: "19de7b34-20ec-4840-8901-96b1c50c1964") : secret "image-registry-tls" not found
May 26 16:03:32.128409 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.128370    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19de7b34-20ec-4840-8901-96b1c50c1964-trusted-ca\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.129828 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.129804    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19de7b34-20ec-4840-8901-96b1c50c1964-image-registry-private-configuration\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.129970 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.129949    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19de7b34-20ec-4840-8901-96b1c50c1964-installation-pull-secrets\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.133770 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.133748    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t"
May 26 16:03:32.138033 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.138004    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-bound-sa-token\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.138639 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.138610    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gk92\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-kube-api-access-9gk92\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.140325 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.140304    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:32.167744 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.167716    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-j57x7"]
May 26 16:03:32.172471 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:03:32.172442    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d3526bf_6f30_4cd2_b6a9_44a0dfc32777.slice/crio-447ea20586830301e00727d1cf55224d0639cfb854323eee58a88ce9e83597b6 WatchSource:0}: Error finding container 447ea20586830301e00727d1cf55224d0639cfb854323eee58a88ce9e83597b6: Status 404 returned error can't find the container with id 447ea20586830301e00727d1cf55224d0639cfb854323eee58a88ce9e83597b6
May 26 16:03:32.262140 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.262106    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t"]
May 26 16:03:32.265402 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:03:32.265361    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4f1c38_12f6_4b6c_9360_0ba547d5f28a.slice/crio-fc89bf39bdf93ce86a10fa28097c69113cefe56a994423b619399c10645359b2 WatchSource:0}: Error finding container fc89bf39bdf93ce86a10fa28097c69113cefe56a994423b619399c10645359b2: Status 404 returned error can't find the container with id fc89bf39bdf93ce86a10fa28097c69113cefe56a994423b619399c10645359b2
May 26 16:03:32.278084 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.278032    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-vkhrc"]
May 26 16:03:32.280622 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:03:32.280597    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad1242b_3f03_48c6_a0cb_f029836842ff.slice/crio-8bae5fcaa7f4c535fdffeddec055714266393d16a9da48456425b26c0825a939 WatchSource:0}: Error finding container 8bae5fcaa7f4c535fdffeddec055714266393d16a9da48456425b26c0825a939: Status 404 returned error can't find the container with id 8bae5fcaa7f4c535fdffeddec055714266393d16a9da48456425b26c0825a939
May 26 16:03:32.493544 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.493505    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t" event={"ID":"8f4f1c38-12f6-4b6c-9360-0ba547d5f28a","Type":"ContainerStarted","Data":"fc89bf39bdf93ce86a10fa28097c69113cefe56a994423b619399c10645359b2"}
May 26 16:03:32.494318 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.494299    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-vkhrc" event={"ID":"8ad1242b-3f03-48c6-a0cb-f029836842ff","Type":"ContainerStarted","Data":"8bae5fcaa7f4c535fdffeddec055714266393d16a9da48456425b26c0825a939"}
May 26 16:03:32.495195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.495175    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-j57x7" event={"ID":"7d3526bf-6f30-4cd2-b6a9-44a0dfc32777","Type":"ContainerStarted","Data":"447ea20586830301e00727d1cf55224d0639cfb854323eee58a88ce9e83597b6"}
May 26 16:03:32.530603 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.530547    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-9zl7z\" (UID: \"7028a76e-6db4-4b2c-afd5-d7ab066dc10a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:03:32.530684 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:32.530673    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 16:03:32.530731 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:32.530723    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls podName:7028a76e-6db4-4b2c-afd5-d7ab066dc10a nodeName:}" failed. No retries permitted until 2026-05-26 16:03:33.530710593 +0000 UTC m=+130.954303731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-9zl7z" (UID: "7028a76e-6db4-4b2c-afd5-d7ab066dc10a") : secret "cluster-monitoring-operator-tls" not found
May 26 16:03:32.631502 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.631469    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:32.631634 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:32.631605    2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:03:32.631634 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:32.631619    2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86594ddd4-x9sdj: secret "image-registry-tls" not found
May 26 16:03:32.631757 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:32.631668    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls podName:19de7b34-20ec-4840-8901-96b1c50c1964 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:33.631653455 +0000 UTC m=+131.055246599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls") pod "image-registry-86594ddd4-x9sdj" (UID: "19de7b34-20ec-4840-8901-96b1c50c1964") : secret "image-registry-tls" not found
May 26 16:03:32.935001 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:32.934915    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs\") pod \"network-metrics-daemon-j528d\" (UID: \"22219f09-c007-46a5-b425-df729845bf45\") " pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:03:32.935456 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:32.935097    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 16:03:32.935456 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:32.935172    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs podName:22219f09-c007-46a5-b425-df729845bf45 nodeName:}" failed. No retries permitted until 2026-05-26 16:05:34.935152106 +0000 UTC m=+252.358745247 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs") pod "network-metrics-daemon-j528d" (UID: "22219f09-c007-46a5-b425-df729845bf45") : secret "metrics-daemon-secret" not found
May 26 16:03:33.540026 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:33.539984    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-9zl7z\" (UID: \"7028a76e-6db4-4b2c-afd5-d7ab066dc10a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:03:33.540252 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:33.540110    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 16:03:33.540252 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:33.540177    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls podName:7028a76e-6db4-4b2c-afd5-d7ab066dc10a nodeName:}" failed. No retries permitted until 2026-05-26 16:03:35.540158584 +0000 UTC m=+132.963751727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-9zl7z" (UID: "7028a76e-6db4-4b2c-afd5-d7ab066dc10a") : secret "cluster-monitoring-operator-tls" not found
May 26 16:03:33.641249 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:33.641221    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:33.641420 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:33.641379    2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:03:33.641420 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:33.641413    2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86594ddd4-x9sdj: secret "image-registry-tls" not found
May 26 16:03:33.641505 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:33.641479    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls podName:19de7b34-20ec-4840-8901-96b1c50c1964 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:35.641459312 +0000 UTC m=+133.065052455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls") pod "image-registry-86594ddd4-x9sdj" (UID: "19de7b34-20ec-4840-8901-96b1c50c1964") : secret "image-registry-tls" not found
May 26 16:03:34.500868 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:34.500813    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-j57x7" event={"ID":"7d3526bf-6f30-4cd2-b6a9-44a0dfc32777","Type":"ContainerStarted","Data":"794560254be87867816da1073bc44d2fb6f4f5d5ed37ee1b3e14d0c0bef06f0a"}
May 26 16:03:34.516333 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:34.516280    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-j57x7" podStartSLOduration=2.034277856 podStartE2EDuration="3.516265469s" podCreationTimestamp="2026-05-26 16:03:31 +0000 UTC" firstStartedPulling="2026-05-26 16:03:32.174532434 +0000 UTC m=+129.598125573" lastFinishedPulling="2026-05-26 16:03:33.656520044 +0000 UTC m=+131.080113186" observedRunningTime="2026-05-26 16:03:34.515898397 +0000 UTC m=+131.939491560" watchObservedRunningTime="2026-05-26 16:03:34.516265469 +0000 UTC m=+131.939858629"
May 26 16:03:35.504315 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:35.504286    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/0.log"
May 26 16:03:35.504731 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:35.504327    2560 generic.go:358] "Generic (PLEG): container finished" podID="8ad1242b-3f03-48c6-a0cb-f029836842ff" containerID="b9e6e8e589a934bd42c798a775daf451385de6d40eeaadd00d479414d388daf3" exitCode=255
May 26 16:03:35.504731 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:35.504422    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-vkhrc" event={"ID":"8ad1242b-3f03-48c6-a0cb-f029836842ff","Type":"ContainerDied","Data":"b9e6e8e589a934bd42c798a775daf451385de6d40eeaadd00d479414d388daf3"}
May 26 16:03:35.504731 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:35.504661    2560 scope.go:117] "RemoveContainer" containerID="b9e6e8e589a934bd42c798a775daf451385de6d40eeaadd00d479414d388daf3"
May 26 16:03:35.505805 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:35.505777    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t" event={"ID":"8f4f1c38-12f6-4b6c-9360-0ba547d5f28a","Type":"ContainerStarted","Data":"18489eee4475ce4bf257cf2c98443bbb8588039e8b4b96231db858282bb20a90"}
May 26 16:03:35.557645 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:35.557614    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-9zl7z\" (UID: \"7028a76e-6db4-4b2c-afd5-d7ab066dc10a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:03:35.557787 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:35.557767    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 16:03:35.557842 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:35.557835    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls podName:7028a76e-6db4-4b2c-afd5-d7ab066dc10a nodeName:}" failed. No retries permitted until 2026-05-26 16:03:39.557815174 +0000 UTC m=+136.981408317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-9zl7z" (UID: "7028a76e-6db4-4b2c-afd5-d7ab066dc10a") : secret "cluster-monitoring-operator-tls" not found
May 26 16:03:35.658696 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:35.658661    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:35.658843 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:35.658767    2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:03:35.658843 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:35.658785    2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86594ddd4-x9sdj: secret "image-registry-tls" not found
May 26 16:03:35.658954 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:35.658848    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls podName:19de7b34-20ec-4840-8901-96b1c50c1964 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:39.658826836 +0000 UTC m=+137.082419991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls") pod "image-registry-86594ddd4-x9sdj" (UID: "19de7b34-20ec-4840-8901-96b1c50c1964") : secret "image-registry-tls" not found
May 26 16:03:35.955543 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:35.955432    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t" podStartSLOduration=2.507090218 podStartE2EDuration="4.955409454s" podCreationTimestamp="2026-05-26 16:03:31 +0000 UTC" firstStartedPulling="2026-05-26 16:03:32.267334915 +0000 UTC m=+129.690928055" lastFinishedPulling="2026-05-26 16:03:34.715654153 +0000 UTC m=+132.139247291" observedRunningTime="2026-05-26 16:03:35.533946701 +0000 UTC m=+132.957539867" watchObservedRunningTime="2026-05-26 16:03:35.955409454 +0000 UTC m=+133.379002619"
May 26 16:03:35.955963 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:35.955944    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-fjj8k"]
May 26 16:03:35.957772 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:35.957757    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fjj8k"
May 26 16:03:35.959943 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:35.959924    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
May 26 16:03:35.960091 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:35.960073    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
May 26 16:03:35.960314 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:35.960300    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ll5mg\""
May 26 16:03:35.967149 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:35.967129    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-fjj8k"]
May 26 16:03:36.061258 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:36.061235    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkvt8\" (UniqueName: \"kubernetes.io/projected/f4b93dc5-3aee-4241-80be-623c0af940da-kube-api-access-lkvt8\") pod \"migrator-b985f5d89-fjj8k\" (UID: \"f4b93dc5-3aee-4241-80be-623c0af940da\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fjj8k"
May 26 16:03:36.161621 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:36.161594    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lkvt8\" (UniqueName: \"kubernetes.io/projected/f4b93dc5-3aee-4241-80be-623c0af940da-kube-api-access-lkvt8\") pod \"migrator-b985f5d89-fjj8k\" (UID: \"f4b93dc5-3aee-4241-80be-623c0af940da\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fjj8k"
May 26 16:03:36.169820 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:36.169796    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkvt8\" (UniqueName: \"kubernetes.io/projected/f4b93dc5-3aee-4241-80be-623c0af940da-kube-api-access-lkvt8\") pod \"migrator-b985f5d89-fjj8k\" (UID: \"f4b93dc5-3aee-4241-80be-623c0af940da\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fjj8k"
May 26 16:03:36.266758 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:36.266737    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fjj8k"
May 26 16:03:36.385058 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:36.385029    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-fjj8k"]
May 26 16:03:36.387700 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:03:36.387670    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b93dc5_3aee_4241_80be_623c0af940da.slice/crio-381df04f859a16fde27568b7e7c0934e973474c01ee7f9d83d0fb67f2a1d4642 WatchSource:0}: Error finding container 381df04f859a16fde27568b7e7c0934e973474c01ee7f9d83d0fb67f2a1d4642: Status 404 returned error can't find the container with id 381df04f859a16fde27568b7e7c0934e973474c01ee7f9d83d0fb67f2a1d4642
May 26 16:03:36.509098 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:36.509069    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:03:36.509512 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:36.509496    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/0.log"
May 26 16:03:36.509649 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:36.509531    2560 generic.go:358] "Generic (PLEG): container finished" podID="8ad1242b-3f03-48c6-a0cb-f029836842ff" containerID="38aff7c83a3178a7ef7f86b912b16a97c1fb33ee6e2c7d288b76cd2c64f10727" exitCode=255
May 26 16:03:36.509649 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:36.509595    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-vkhrc" event={"ID":"8ad1242b-3f03-48c6-a0cb-f029836842ff","Type":"ContainerDied","Data":"38aff7c83a3178a7ef7f86b912b16a97c1fb33ee6e2c7d288b76cd2c64f10727"}
May 26 16:03:36.509649 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:36.509635    2560 scope.go:117] "RemoveContainer" containerID="b9e6e8e589a934bd42c798a775daf451385de6d40eeaadd00d479414d388daf3"
May 26 16:03:36.509885 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:36.509867    2560 scope.go:117] "RemoveContainer" containerID="38aff7c83a3178a7ef7f86b912b16a97c1fb33ee6e2c7d288b76cd2c64f10727"
May 26 16:03:36.510088 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:36.510065    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-vkhrc_openshift-console-operator(8ad1242b-3f03-48c6-a0cb-f029836842ff)\"" pod="openshift-console-operator/console-operator-76dff48d59-vkhrc" podUID="8ad1242b-3f03-48c6-a0cb-f029836842ff"
May 26 16:03:36.510715 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:36.510696    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fjj8k" event={"ID":"f4b93dc5-3aee-4241-80be-623c0af940da","Type":"ContainerStarted","Data":"381df04f859a16fde27568b7e7c0934e973474c01ee7f9d83d0fb67f2a1d4642"}
May 26 16:03:37.514571 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:37.514544    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:03:37.515077 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:37.514957    2560 scope.go:117] "RemoveContainer" containerID="38aff7c83a3178a7ef7f86b912b16a97c1fb33ee6e2c7d288b76cd2c64f10727"
May 26 16:03:37.515188 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:37.515166    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-vkhrc_openshift-console-operator(8ad1242b-3f03-48c6-a0cb-f029836842ff)\"" pod="openshift-console-operator/console-operator-76dff48d59-vkhrc" podUID="8ad1242b-3f03-48c6-a0cb-f029836842ff"
May 26 16:03:38.374404 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.374360    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-bzqlz"]
May 26 16:03:38.376149 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.376134    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-bzqlz"
May 26 16:03:38.378965 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.378919    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
May 26 16:03:38.378965 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.378941    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
May 26 16:03:38.379156 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.379141    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
May 26 16:03:38.379986 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.379968    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
May 26 16:03:38.380076 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.380058    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xgl8g\""
May 26 16:03:38.390524 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.390502    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-bzqlz"]
May 26 16:03:38.480652 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.480620    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/13186435-a27d-47ba-b29d-425dda3bc434-signing-cabundle\") pod \"service-ca-67d667c7d-bzqlz\" (UID: \"13186435-a27d-47ba-b29d-425dda3bc434\") " pod="openshift-service-ca/service-ca-67d667c7d-bzqlz"
May 26 16:03:38.480816 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.480678    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfr5x\" (UniqueName: \"kubernetes.io/projected/13186435-a27d-47ba-b29d-425dda3bc434-kube-api-access-zfr5x\") pod \"service-ca-67d667c7d-bzqlz\" (UID: \"13186435-a27d-47ba-b29d-425dda3bc434\") " pod="openshift-service-ca/service-ca-67d667c7d-bzqlz"
May 26 16:03:38.480816 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.480788    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/13186435-a27d-47ba-b29d-425dda3bc434-signing-key\") pod \"service-ca-67d667c7d-bzqlz\" (UID: \"13186435-a27d-47ba-b29d-425dda3bc434\") " pod="openshift-service-ca/service-ca-67d667c7d-bzqlz"
May 26 16:03:38.519305 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.519268    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fjj8k" event={"ID":"f4b93dc5-3aee-4241-80be-623c0af940da","Type":"ContainerStarted","Data":"34d3958b82c200666928bcd07f8c0619eba3078094b9482420442877dd4dd60f"}
May 26 16:03:38.519305 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.519302    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fjj8k" event={"ID":"f4b93dc5-3aee-4241-80be-623c0af940da","Type":"ContainerStarted","Data":"285ccaad7fdbb855c8c8b95028650d4165de63dd8fb3b9aaf2a247110e876baa"}
May 26 16:03:38.535306 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.535262    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fjj8k" podStartSLOduration=2.288792972 podStartE2EDuration="3.535250207s" podCreationTimestamp="2026-05-26 16:03:35 +0000 UTC" firstStartedPulling="2026-05-26 16:03:36.3893741 +0000 UTC m=+133.812967239" lastFinishedPulling="2026-05-26 16:03:37.635831335 +0000 UTC m=+135.059424474" observedRunningTime="2026-05-26 16:03:38.534488197 +0000 UTC m=+135.958081361" watchObservedRunningTime="2026-05-26 16:03:38.535250207 +0000 UTC m=+135.958843365"
May 26 16:03:38.581914 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.581889    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/13186435-a27d-47ba-b29d-425dda3bc434-signing-cabundle\") pod \"service-ca-67d667c7d-bzqlz\" (UID: \"13186435-a27d-47ba-b29d-425dda3bc434\") " pod="openshift-service-ca/service-ca-67d667c7d-bzqlz"
May 26 16:03:38.582022 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.581936    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zfr5x\" (UniqueName: \"kubernetes.io/projected/13186435-a27d-47ba-b29d-425dda3bc434-kube-api-access-zfr5x\") pod \"service-ca-67d667c7d-bzqlz\" (UID: \"13186435-a27d-47ba-b29d-425dda3bc434\") " pod="openshift-service-ca/service-ca-67d667c7d-bzqlz"
May 26 16:03:38.582022 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.581984    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/13186435-a27d-47ba-b29d-425dda3bc434-signing-key\") pod \"service-ca-67d667c7d-bzqlz\" (UID: \"13186435-a27d-47ba-b29d-425dda3bc434\") " pod="openshift-service-ca/service-ca-67d667c7d-bzqlz"
May 26 16:03:38.582523 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.582498    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/13186435-a27d-47ba-b29d-425dda3bc434-signing-cabundle\") pod \"service-ca-67d667c7d-bzqlz\" (UID: \"13186435-a27d-47ba-b29d-425dda3bc434\") " pod="openshift-service-ca/service-ca-67d667c7d-bzqlz"
May 26 16:03:38.584290 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.584267    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/13186435-a27d-47ba-b29d-425dda3bc434-signing-key\") pod \"service-ca-67d667c7d-bzqlz\" (UID: \"13186435-a27d-47ba-b29d-425dda3bc434\") " pod="openshift-service-ca/service-ca-67d667c7d-bzqlz"
May 26 16:03:38.589947 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.589923    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfr5x\" (UniqueName: \"kubernetes.io/projected/13186435-a27d-47ba-b29d-425dda3bc434-kube-api-access-zfr5x\") pod \"service-ca-67d667c7d-bzqlz\" (UID: \"13186435-a27d-47ba-b29d-425dda3bc434\") " pod="openshift-service-ca/service-ca-67d667c7d-bzqlz"
May 26 16:03:38.684463 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.684372    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-bzqlz"
May 26 16:03:38.802558 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:38.802528    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-bzqlz"]
May 26 16:03:38.805613 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:03:38.805589    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13186435_a27d_47ba_b29d_425dda3bc434.slice/crio-cd9ad1e8be68b8fe5a4ec4179dc70d5c5d2b714780b3d6a5a291e475c919359f WatchSource:0}: Error finding container cd9ad1e8be68b8fe5a4ec4179dc70d5c5d2b714780b3d6a5a291e475c919359f: Status 404 returned error can't find the container with id cd9ad1e8be68b8fe5a4ec4179dc70d5c5d2b714780b3d6a5a291e475c919359f
May 26 16:03:39.192773 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:39.192749    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xx9vq_a1e36e32-7353-4703-8b16-848a49a47e91/dns-node-resolver/0.log"
May 26 16:03:39.523067 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:39.523029    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-bzqlz" event={"ID":"13186435-a27d-47ba-b29d-425dda3bc434","Type":"ContainerStarted","Data":"cd9ad1e8be68b8fe5a4ec4179dc70d5c5d2b714780b3d6a5a291e475c919359f"}
May 26 16:03:39.589424 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:39.589379    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-9zl7z\" (UID: \"7028a76e-6db4-4b2c-afd5-d7ab066dc10a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:03:39.589593 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:39.589542    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 16:03:39.589661 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:39.589603    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls podName:7028a76e-6db4-4b2c-afd5-d7ab066dc10a nodeName:}" failed. No retries permitted until 2026-05-26 16:03:47.58958691 +0000 UTC m=+145.013180057 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-9zl7z" (UID: "7028a76e-6db4-4b2c-afd5-d7ab066dc10a") : secret "cluster-monitoring-operator-tls" not found
May 26 16:03:39.690421 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:39.690376    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:39.690584 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:39.690534    2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:03:39.690584 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:39.690554    2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86594ddd4-x9sdj: secret "image-registry-tls" not found
May 26 16:03:39.690692 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:39.690611    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls podName:19de7b34-20ec-4840-8901-96b1c50c1964 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:47.690591357 +0000 UTC m=+145.114184495 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls") pod "image-registry-86594ddd4-x9sdj" (UID: "19de7b34-20ec-4840-8901-96b1c50c1964") : secret "image-registry-tls" not found
May 26 16:03:39.991623 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:39.991600    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mz8tq_ea5226b3-f441-403d-aa44-8ac33afd311b/node-ca/0.log"
May 26 16:03:41.195663 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:41.195632    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-fjj8k_f4b93dc5-3aee-4241-80be-623c0af940da/migrator/0.log"
May 26 16:03:41.391447 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:41.391410    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-fjj8k_f4b93dc5-3aee-4241-80be-623c0af940da/graceful-termination/0.log"
May 26 16:03:41.536224 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:41.536174    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-bzqlz" event={"ID":"13186435-a27d-47ba-b29d-425dda3bc434","Type":"ContainerStarted","Data":"220daafbfb8d0f10661ad6c5f9b6d6d0b3c5e8c79edf1e780246591f401f7423"}
May 26 16:03:41.555383 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:41.555074    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-bzqlz" podStartSLOduration=1.6794727379999999 podStartE2EDuration="3.555059313s" podCreationTimestamp="2026-05-26 16:03:38 +0000 UTC" firstStartedPulling="2026-05-26 16:03:38.807455838 +0000 UTC m=+136.231048981" lastFinishedPulling="2026-05-26 16:03:40.683042403 +0000 UTC m=+138.106635556" observedRunningTime="2026-05-26 16:03:41.554939003 +0000 UTC m=+138.978532164" watchObservedRunningTime="2026-05-26 16:03:41.555059313 +0000 UTC m=+138.978652497"
May 26 16:03:41.592549 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:41.592469    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-mmd9t_8f4f1c38-12f6-4b6c-9360-0ba547d5f28a/kube-storage-version-migrator-operator/0.log"
May 26 16:03:42.140843 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:42.140805    2560 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:42.140843 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:42.140843    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:42.141295 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:42.141275    2560 scope.go:117] "RemoveContainer" containerID="38aff7c83a3178a7ef7f86b912b16a97c1fb33ee6e2c7d288b76cd2c64f10727"
May 26 16:03:42.141529 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:42.141507    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-vkhrc_openshift-console-operator(8ad1242b-3f03-48c6-a0cb-f029836842ff)\"" pod="openshift-console-operator/console-operator-76dff48d59-vkhrc" podUID="8ad1242b-3f03-48c6-a0cb-f029836842ff"
May 26 16:03:47.651168 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:47.651133    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-9zl7z\" (UID: \"7028a76e-6db4-4b2c-afd5-d7ab066dc10a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:03:47.651650 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:47.651245    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 16:03:47.651650 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:03:47.651296    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls podName:7028a76e-6db4-4b2c-afd5-d7ab066dc10a nodeName:}" failed. No retries permitted until 2026-05-26 16:04:03.651281014 +0000 UTC m=+161.074874154 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-9zl7z" (UID: "7028a76e-6db4-4b2c-afd5-d7ab066dc10a") : secret "cluster-monitoring-operator-tls" not found
May 26 16:03:47.752137 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:47.752108    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:47.754427 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:47.754409    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls\") pod \"image-registry-86594ddd4-x9sdj\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") " pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:47.854512 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:47.854483    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:47.975735 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:47.975707    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86594ddd4-x9sdj"]
May 26 16:03:47.979113 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:03:47.979087    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19de7b34_20ec_4840_8901_96b1c50c1964.slice/crio-729d75bd6b4fe87821048edf67eaf11beb71c5218b6df180624dbc5445c76af4 WatchSource:0}: Error finding container 729d75bd6b4fe87821048edf67eaf11beb71c5218b6df180624dbc5445c76af4: Status 404 returned error can't find the container with id 729d75bd6b4fe87821048edf67eaf11beb71c5218b6df180624dbc5445c76af4
May 26 16:03:48.556382 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:48.556343    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86594ddd4-x9sdj" event={"ID":"19de7b34-20ec-4840-8901-96b1c50c1964","Type":"ContainerStarted","Data":"963b364b9ec6408d84bc39207aabf76de60154551140bf193c8c986a4f0f5358"}
May 26 16:03:48.556382 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:48.556385    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86594ddd4-x9sdj" event={"ID":"19de7b34-20ec-4840-8901-96b1c50c1964","Type":"ContainerStarted","Data":"729d75bd6b4fe87821048edf67eaf11beb71c5218b6df180624dbc5445c76af4"}
May 26 16:03:48.556630 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:48.556588    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:03:48.579771 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:48.579726    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86594ddd4-x9sdj" podStartSLOduration=17.579714391 podStartE2EDuration="17.579714391s" podCreationTimestamp="2026-05-26 16:03:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:03:48.578618014 +0000 UTC m=+146.002211176" watchObservedRunningTime="2026-05-26 16:03:48.579714391 +0000 UTC m=+146.003307572"
May 26 16:03:56.166748 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:56.166717    2560 scope.go:117] "RemoveContainer" containerID="38aff7c83a3178a7ef7f86b912b16a97c1fb33ee6e2c7d288b76cd2c64f10727"
May 26 16:03:56.576700 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:56.576676    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:03:56.576883 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:56.576752    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-vkhrc" event={"ID":"8ad1242b-3f03-48c6-a0cb-f029836842ff","Type":"ContainerStarted","Data":"34bf2aa037104df6d41e21248d328038da229a08f7a731a6d1a06228c21c81f4"}
May 26 16:03:56.577037 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:56.577019    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:03:56.593897 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:56.593836    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-vkhrc" podStartSLOduration=23.162088321 podStartE2EDuration="25.59382243s" podCreationTimestamp="2026-05-26 16:03:31 +0000 UTC" firstStartedPulling="2026-05-26 16:03:32.282237871 +0000 UTC m=+129.705831011" lastFinishedPulling="2026-05-26 16:03:34.713971968 +0000 UTC m=+132.137565120" observedRunningTime="2026-05-26 16:03:56.592650164 +0000 UTC m=+154.016243339" watchObservedRunningTime="2026-05-26 16:03:56.59382243 +0000 UTC m=+154.017415590"
May 26 16:03:57.577260 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:57.577225    2560 patch_prober.go:28] interesting pod/console-operator-76dff48d59-vkhrc container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.134.0.9:8443/readyz\": context deadline exceeded" start-of-body=
May 26 16:03:57.577730 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:57.577279    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-76dff48d59-vkhrc" podUID="8ad1242b-3f03-48c6-a0cb-f029836842ff" containerName="console-operator" probeResult="failure" output="Get \"https://10.134.0.9:8443/readyz\": context deadline exceeded"
May 26 16:03:57.651205 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:03:57.651178    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-vkhrc"
May 26 16:04:00.344338 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.344298    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86594ddd4-x9sdj"]
May 26 16:04:00.378471 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.378441    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ql8x6"]
May 26 16:04:00.381564 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.381545    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.384196 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.384179    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 26 16:04:00.384291 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.384184    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 26 16:04:00.384291 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.384224    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 26 16:04:00.384622 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.384609    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 26 16:04:00.384850 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.384835    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vtbnj\""
May 26 16:04:00.392967 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.392948    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ql8x6"]
May 26 16:04:00.449094 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.449065    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/517f1a5d-7656-4456-846f-87eb3f4885e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ql8x6\" (UID: \"517f1a5d-7656-4456-846f-87eb3f4885e6\") " pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.449232 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.449188    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlh5b\" (UniqueName: \"kubernetes.io/projected/517f1a5d-7656-4456-846f-87eb3f4885e6-kube-api-access-qlh5b\") pod \"insights-runtime-extractor-ql8x6\" (UID: \"517f1a5d-7656-4456-846f-87eb3f4885e6\") " pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.449292 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.449231    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/517f1a5d-7656-4456-846f-87eb3f4885e6-data-volume\") pod \"insights-runtime-extractor-ql8x6\" (UID: \"517f1a5d-7656-4456-846f-87eb3f4885e6\") " pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.449292 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.449252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/517f1a5d-7656-4456-846f-87eb3f4885e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ql8x6\" (UID: \"517f1a5d-7656-4456-846f-87eb3f4885e6\") " pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.449375 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.449294    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/517f1a5d-7656-4456-846f-87eb3f4885e6-crio-socket\") pod \"insights-runtime-extractor-ql8x6\" (UID: \"517f1a5d-7656-4456-846f-87eb3f4885e6\") " pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.549839 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.549816    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qlh5b\" (UniqueName: \"kubernetes.io/projected/517f1a5d-7656-4456-846f-87eb3f4885e6-kube-api-access-qlh5b\") pod \"insights-runtime-extractor-ql8x6\" (UID: \"517f1a5d-7656-4456-846f-87eb3f4885e6\") " pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.549949 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.549854    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/517f1a5d-7656-4456-846f-87eb3f4885e6-data-volume\") pod \"insights-runtime-extractor-ql8x6\" (UID: \"517f1a5d-7656-4456-846f-87eb3f4885e6\") " pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.549949 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.549870    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/517f1a5d-7656-4456-846f-87eb3f4885e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ql8x6\" (UID: \"517f1a5d-7656-4456-846f-87eb3f4885e6\") " pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.549949 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.549887    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/517f1a5d-7656-4456-846f-87eb3f4885e6-crio-socket\") pod \"insights-runtime-extractor-ql8x6\" (UID: \"517f1a5d-7656-4456-846f-87eb3f4885e6\") " pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.549949 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.549937    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/517f1a5d-7656-4456-846f-87eb3f4885e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ql8x6\" (UID: \"517f1a5d-7656-4456-846f-87eb3f4885e6\") " pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.550096 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.550039    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/517f1a5d-7656-4456-846f-87eb3f4885e6-crio-socket\") pod \"insights-runtime-extractor-ql8x6\" (UID: \"517f1a5d-7656-4456-846f-87eb3f4885e6\") " pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.550235 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.550219    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/517f1a5d-7656-4456-846f-87eb3f4885e6-data-volume\") pod \"insights-runtime-extractor-ql8x6\" (UID: \"517f1a5d-7656-4456-846f-87eb3f4885e6\") " pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.550400 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.550362    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/517f1a5d-7656-4456-846f-87eb3f4885e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ql8x6\" (UID: \"517f1a5d-7656-4456-846f-87eb3f4885e6\") " pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.552100 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.552076    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/517f1a5d-7656-4456-846f-87eb3f4885e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ql8x6\" (UID: \"517f1a5d-7656-4456-846f-87eb3f4885e6\") " pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.557937 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.557914    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlh5b\" (UniqueName: \"kubernetes.io/projected/517f1a5d-7656-4456-846f-87eb3f4885e6-kube-api-access-qlh5b\") pod \"insights-runtime-extractor-ql8x6\" (UID: \"517f1a5d-7656-4456-846f-87eb3f4885e6\") " pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.689829 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.689758    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ql8x6"
May 26 16:04:00.808379 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:00.808346    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ql8x6"]
May 26 16:04:00.811875 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:04:00.811840    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517f1a5d_7656_4456_846f_87eb3f4885e6.slice/crio-0ebd9075b19d04af59e409c9ad7ef0dbcf780d1b4723810772c76ad503afb427 WatchSource:0}: Error finding container 0ebd9075b19d04af59e409c9ad7ef0dbcf780d1b4723810772c76ad503afb427: Status 404 returned error can't find the container with id 0ebd9075b19d04af59e409c9ad7ef0dbcf780d1b4723810772c76ad503afb427
May 26 16:04:01.589639 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:01.589601    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ql8x6" event={"ID":"517f1a5d-7656-4456-846f-87eb3f4885e6","Type":"ContainerStarted","Data":"a6ba9faad8ec9a061ff55976dd54d1d3aad913cbcdaf46ef0c7067e9d8e66a7a"}
May 26 16:04:01.589962 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:01.589646    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ql8x6" event={"ID":"517f1a5d-7656-4456-846f-87eb3f4885e6","Type":"ContainerStarted","Data":"0ebd9075b19d04af59e409c9ad7ef0dbcf780d1b4723810772c76ad503afb427"}
May 26 16:04:02.594098 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:02.594061    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ql8x6" event={"ID":"517f1a5d-7656-4456-846f-87eb3f4885e6","Type":"ContainerStarted","Data":"ced69105dcf3a5794077cc437bbf070ca3990b32381c9273dc34505e25b92666"}
May 26 16:04:03.518898 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:04:03.518849    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-v7sq4" podUID="b7627daa-7d0e-4a02-9b00-a3515e6b7137"
May 26 16:04:03.538018 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:04:03.537984    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-29s87" podUID="fcbf8b71-6b16-4832-b58b-537691709f83"
May 26 16:04:03.597773 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:03.597740    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ql8x6" event={"ID":"517f1a5d-7656-4456-846f-87eb3f4885e6","Type":"ContainerStarted","Data":"00c231e93cda7c52eb910efea38d26b7422c11979fbee06fe5c471cacf19c2c7"}
May 26 16:04:03.597773 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:03.597772    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v7sq4"
May 26 16:04:03.616314 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:03.616264    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ql8x6" podStartSLOduration=1.485956599 podStartE2EDuration="3.616250774s" podCreationTimestamp="2026-05-26 16:04:00 +0000 UTC" firstStartedPulling="2026-05-26 16:04:00.872531633 +0000 UTC m=+158.296124785" lastFinishedPulling="2026-05-26 16:04:03.002825815 +0000 UTC m=+160.426418960" observedRunningTime="2026-05-26 16:04:03.61564115 +0000 UTC m=+161.039234311" watchObservedRunningTime="2026-05-26 16:04:03.616250774 +0000 UTC m=+161.039843934"
May 26 16:04:03.676787 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:03.676745    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-9zl7z\" (UID: \"7028a76e-6db4-4b2c-afd5-d7ab066dc10a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:04:03.679240 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:03.679213    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7028a76e-6db4-4b2c-afd5-d7ab066dc10a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-9zl7z\" (UID: \"7028a76e-6db4-4b2c-afd5-d7ab066dc10a\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:04:03.946354 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:03.946267    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"
May 26 16:04:04.064505 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:04.064475    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z"]
May 26 16:04:04.067354 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:04:04.067325    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7028a76e_6db4_4b2c_afd5_d7ab066dc10a.slice/crio-1945a4dd5fdd58f99bbd19449edeff5093f0d76588add5e3ee72e9f9edb444fc WatchSource:0}: Error finding container 1945a4dd5fdd58f99bbd19449edeff5093f0d76588add5e3ee72e9f9edb444fc: Status 404 returned error can't find the container with id 1945a4dd5fdd58f99bbd19449edeff5093f0d76588add5e3ee72e9f9edb444fc
May 26 16:04:04.176259 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:04:04.176218    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-j528d" podUID="22219f09-c007-46a5-b425-df729845bf45"
May 26 16:04:04.601004 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:04.600927    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z" event={"ID":"7028a76e-6db4-4b2c-afd5-d7ab066dc10a","Type":"ContainerStarted","Data":"1945a4dd5fdd58f99bbd19449edeff5093f0d76588add5e3ee72e9f9edb444fc"}
May 26 16:04:06.227738 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:06.227661    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9"]
May 26 16:04:06.229842 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:06.229822    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9"
May 26 16:04:06.232124 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:06.232105    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 26 16:04:06.232230 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:06.232156    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hrv8w\""
May 26 16:04:06.242494 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:06.242474    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9"]
May 26 16:04:06.297797 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:06.297770    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/533d5f01-445b-4e8e-a3be-7acc89343863-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9\" (UID: \"533d5f01-445b-4e8e-a3be-7acc89343863\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9"
May 26 16:04:06.398150 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:06.398123    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/533d5f01-445b-4e8e-a3be-7acc89343863-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9\" (UID: \"533d5f01-445b-4e8e-a3be-7acc89343863\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9"
May 26 16:04:06.398261 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:04:06.398235    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
May 26 16:04:06.398325 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:04:06.398315    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533d5f01-445b-4e8e-a3be-7acc89343863-tls-certificates podName:533d5f01-445b-4e8e-a3be-7acc89343863 nodeName:}" failed. No retries permitted until 2026-05-26 16:04:06.898277649 +0000 UTC m=+164.321870788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/533d5f01-445b-4e8e-a3be-7acc89343863-tls-certificates") pod "prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9" (UID: "533d5f01-445b-4e8e-a3be-7acc89343863") : secret "prometheus-operator-admission-webhook-tls" not found
May 26 16:04:06.607900 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:06.607867    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z" event={"ID":"7028a76e-6db4-4b2c-afd5-d7ab066dc10a","Type":"ContainerStarted","Data":"06cecf1b0d95124b7c2850d57d4d7091a72868909b86548aa22c67541b14f837"}
May 26 16:04:06.624767 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:06.624721    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-9zl7z" podStartSLOduration=33.940366129 podStartE2EDuration="35.624707411s" podCreationTimestamp="2026-05-26 16:03:31 +0000 UTC" firstStartedPulling="2026-05-26 16:04:04.069033191 +0000 UTC m=+161.492626329" lastFinishedPulling="2026-05-26 16:04:05.753374459 +0000 UTC m=+163.176967611" observedRunningTime="2026-05-26 16:04:06.623563267 +0000 UTC m=+164.047156428" watchObservedRunningTime="2026-05-26 16:04:06.624707411 +0000 UTC m=+164.048300598"
May 26 16:04:06.901422 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:06.901332    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/533d5f01-445b-4e8e-a3be-7acc89343863-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9\" (UID: \"533d5f01-445b-4e8e-a3be-7acc89343863\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9"
May 26 16:04:06.903678 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:06.903653    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/533d5f01-445b-4e8e-a3be-7acc89343863-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9\" (UID: \"533d5f01-445b-4e8e-a3be-7acc89343863\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9"
May 26 16:04:07.138283 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:07.138250    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9"
May 26 16:04:07.276294 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:07.276264    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9"]
May 26 16:04:07.278951 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:04:07.278921    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533d5f01_445b_4e8e_a3be_7acc89343863.slice/crio-9d0d7b54e75f1b13257b66c1b78063b758c41e762e8c227c5e9a56b49260f43e WatchSource:0}: Error finding container 9d0d7b54e75f1b13257b66c1b78063b758c41e762e8c227c5e9a56b49260f43e: Status 404 returned error can't find the container with id 9d0d7b54e75f1b13257b66c1b78063b758c41e762e8c227c5e9a56b49260f43e
May 26 16:04:07.611270 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:07.611238    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9" event={"ID":"533d5f01-445b-4e8e-a3be-7acc89343863","Type":"ContainerStarted","Data":"9d0d7b54e75f1b13257b66c1b78063b758c41e762e8c227c5e9a56b49260f43e"}
May 26 16:04:08.411542 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:08.411513    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:04:08.413674 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:08.413653    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7627daa-7d0e-4a02-9b00-a3515e6b7137-metrics-tls\") pod \"dns-default-v7sq4\" (UID: \"b7627daa-7d0e-4a02-9b00-a3515e6b7137\") " pod="openshift-dns/dns-default-v7sq4"
May 26 16:04:08.512150 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:08.512117    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert\") pod \"ingress-canary-29s87\" (UID: \"fcbf8b71-6b16-4832-b58b-537691709f83\") " pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:04:08.514480 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:08.514460    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcbf8b71-6b16-4832-b58b-537691709f83-cert\") pod \"ingress-canary-29s87\" (UID: \"fcbf8b71-6b16-4832-b58b-537691709f83\") " pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:04:08.614612 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:08.614550    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9" event={"ID":"533d5f01-445b-4e8e-a3be-7acc89343863","Type":"ContainerStarted","Data":"98220c72a34164f9cadc3bbb9298de4b08f6fd3089be178893b6f45f7c876aec"}
May 26 16:04:08.614766 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:08.614745    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9"
May 26 16:04:08.619109 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:08.619085    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9"
May 26 16:04:08.629096 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:08.629061    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9" podStartSLOduration=1.599464904 podStartE2EDuration="2.629049722s" podCreationTimestamp="2026-05-26 16:04:06 +0000 UTC" firstStartedPulling="2026-05-26 16:04:07.280964051 +0000 UTC m=+164.704557191" lastFinishedPulling="2026-05-26 16:04:08.31054887 +0000 UTC m=+165.734142009" observedRunningTime="2026-05-26 16:04:08.628294955 +0000 UTC m=+166.051888116" watchObservedRunningTime="2026-05-26 16:04:08.629049722 +0000 UTC m=+166.052642883"
May 26 16:04:08.701037 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:08.701010    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lppkw\""
May 26 16:04:08.709800 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:08.709782    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v7sq4"
May 26 16:04:08.826998 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:08.826972    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v7sq4"]
May 26 16:04:08.829942 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:04:08.829916    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7627daa_7d0e_4a02_9b00_a3515e6b7137.slice/crio-be3849cbbf20a0198724ecd5e36ff7a24b9cf1da6feba1af6271595f3424a5d4 WatchSource:0}: Error finding container be3849cbbf20a0198724ecd5e36ff7a24b9cf1da6feba1af6271595f3424a5d4: Status 404 returned error can't find the container with id be3849cbbf20a0198724ecd5e36ff7a24b9cf1da6feba1af6271595f3424a5d4
May 26 16:04:09.309848 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.309814    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"]
May 26 16:04:09.314873 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.314844    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"
May 26 16:04:09.317371 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.317345    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 26 16:04:09.318634 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.318607    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8jqf4\""
May 26 16:04:09.318634 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.318628    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 26 16:04:09.318807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.318660    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 26 16:04:09.322678 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.322660    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"]
May 26 16:04:09.418252 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.418222    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1839dcf6-a640-4f0c-9bc1-e0258415c54b-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-zbhlt\" (UID: \"1839dcf6-a640-4f0c-9bc1-e0258415c54b\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"
May 26 16:04:09.418692 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.418273    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1839dcf6-a640-4f0c-9bc1-e0258415c54b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-zbhlt\" (UID: \"1839dcf6-a640-4f0c-9bc1-e0258415c54b\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"
May 26 16:04:09.418692 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.418436    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1839dcf6-a640-4f0c-9bc1-e0258415c54b-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-zbhlt\" (UID: \"1839dcf6-a640-4f0c-9bc1-e0258415c54b\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"
May 26 16:04:09.418692 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.418487    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvdw7\" (UniqueName: \"kubernetes.io/projected/1839dcf6-a640-4f0c-9bc1-e0258415c54b-kube-api-access-jvdw7\") pod \"prometheus-operator-7779bbdd78-zbhlt\" (UID: \"1839dcf6-a640-4f0c-9bc1-e0258415c54b\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"
May 26 16:04:09.519854 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.519825    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1839dcf6-a640-4f0c-9bc1-e0258415c54b-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-zbhlt\" (UID: \"1839dcf6-a640-4f0c-9bc1-e0258415c54b\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"
May 26 16:04:09.520028 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.519865    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jvdw7\" (UniqueName: \"kubernetes.io/projected/1839dcf6-a640-4f0c-9bc1-e0258415c54b-kube-api-access-jvdw7\") pod \"prometheus-operator-7779bbdd78-zbhlt\" (UID: \"1839dcf6-a640-4f0c-9bc1-e0258415c54b\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"
May 26 16:04:09.520028 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.519906    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1839dcf6-a640-4f0c-9bc1-e0258415c54b-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-zbhlt\" (UID: \"1839dcf6-a640-4f0c-9bc1-e0258415c54b\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"
May 26 16:04:09.520028 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.519944    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1839dcf6-a640-4f0c-9bc1-e0258415c54b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-zbhlt\" (UID: \"1839dcf6-a640-4f0c-9bc1-e0258415c54b\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"
May 26 16:04:09.520028 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:04:09.519986    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
May 26 16:04:09.520238 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:04:09.520049    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1839dcf6-a640-4f0c-9bc1-e0258415c54b-prometheus-operator-tls podName:1839dcf6-a640-4f0c-9bc1-e0258415c54b nodeName:}" failed. No retries permitted until 2026-05-26 16:04:10.020029419 +0000 UTC m=+167.443622558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/1839dcf6-a640-4f0c-9bc1-e0258415c54b-prometheus-operator-tls") pod "prometheus-operator-7779bbdd78-zbhlt" (UID: "1839dcf6-a640-4f0c-9bc1-e0258415c54b") : secret "prometheus-operator-tls" not found
May 26 16:04:09.520671 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.520648    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1839dcf6-a640-4f0c-9bc1-e0258415c54b-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-zbhlt\" (UID: \"1839dcf6-a640-4f0c-9bc1-e0258415c54b\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"
May 26 16:04:09.522641 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.522618    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1839dcf6-a640-4f0c-9bc1-e0258415c54b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-zbhlt\" (UID: \"1839dcf6-a640-4f0c-9bc1-e0258415c54b\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"
May 26 16:04:09.529446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.529421    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvdw7\" (UniqueName: \"kubernetes.io/projected/1839dcf6-a640-4f0c-9bc1-e0258415c54b-kube-api-access-jvdw7\") pod \"prometheus-operator-7779bbdd78-zbhlt\" (UID: \"1839dcf6-a640-4f0c-9bc1-e0258415c54b\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"
May 26 16:04:09.618335 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:09.618261    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v7sq4" event={"ID":"b7627daa-7d0e-4a02-9b00-a3515e6b7137","Type":"ContainerStarted","Data":"be3849cbbf20a0198724ecd5e36ff7a24b9cf1da6feba1af6271595f3424a5d4"}
May 26 16:04:10.024410 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:10.024360    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1839dcf6-a640-4f0c-9bc1-e0258415c54b-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-zbhlt\" (UID: \"1839dcf6-a640-4f0c-9bc1-e0258415c54b\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"
May 26 16:04:10.027312 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:10.027283    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1839dcf6-a640-4f0c-9bc1-e0258415c54b-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-zbhlt\" (UID: \"1839dcf6-a640-4f0c-9bc1-e0258415c54b\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"
May 26 16:04:10.233816 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:10.233784    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"
May 26 16:04:10.349625 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:10.349593    2560 patch_prober.go:28] interesting pod/image-registry-86594ddd4-x9sdj container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 26 16:04:10.349778 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:10.349642    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-86594ddd4-x9sdj" podUID="19de7b34-20ec-4840-8901-96b1c50c1964" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:04:10.360543 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:10.360515    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt"]
May 26 16:04:10.363781 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:04:10.363758    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1839dcf6_a640_4f0c_9bc1_e0258415c54b.slice/crio-2a63683da73455da4ffd862a2c9ff0773a68f7e646bd129b1f4888113a73f568 WatchSource:0}: Error finding container 2a63683da73455da4ffd862a2c9ff0773a68f7e646bd129b1f4888113a73f568: Status 404 returned error can't find the container with id 2a63683da73455da4ffd862a2c9ff0773a68f7e646bd129b1f4888113a73f568
May 26 16:04:10.622281 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:10.622200    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt" event={"ID":"1839dcf6-a640-4f0c-9bc1-e0258415c54b","Type":"ContainerStarted","Data":"2a63683da73455da4ffd862a2c9ff0773a68f7e646bd129b1f4888113a73f568"}
May 26 16:04:10.623667 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:10.623643    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v7sq4" event={"ID":"b7627daa-7d0e-4a02-9b00-a3515e6b7137","Type":"ContainerStarted","Data":"320a12d0f0e52c8950e1cf01e7fe023fdbabb45601d9c2290dcdc9f131cc9c47"}
May 26 16:04:10.623761 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:10.623673    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v7sq4" event={"ID":"b7627daa-7d0e-4a02-9b00-a3515e6b7137","Type":"ContainerStarted","Data":"4df4436f3c124ea0146b79fcf977ad5b86b75ecd265de356a3f9b3578a6a3cbb"}
May 26 16:04:10.641267 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:10.641228    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-v7sq4" podStartSLOduration=129.354982178 podStartE2EDuration="2m10.641217774s" podCreationTimestamp="2026-05-26 16:02:00 +0000 UTC" firstStartedPulling="2026-05-26 16:04:08.831567892 +0000 UTC m=+166.255161031" lastFinishedPulling="2026-05-26 16:04:10.117803485 +0000 UTC m=+167.541396627" observedRunningTime="2026-05-26 16:04:10.639975566 +0000 UTC m=+168.063568730" watchObservedRunningTime="2026-05-26 16:04:10.641217774 +0000 UTC m=+168.064810934"
May 26 16:04:11.626050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:11.626027    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v7sq4"
May 26 16:04:12.630118 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:12.630083    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt" event={"ID":"1839dcf6-a640-4f0c-9bc1-e0258415c54b","Type":"ContainerStarted","Data":"185e6b98c25d67c671edb85f5da1cb9a4cc00c7169aba176db553aa5d7d456d4"}
May 26 16:04:12.630118 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:12.630118    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt" event={"ID":"1839dcf6-a640-4f0c-9bc1-e0258415c54b","Type":"ContainerStarted","Data":"5f1fd6d1a1311921b070cd2545c8c19af78d94cda6a7580636c7b60f33e9e3b4"}
May 26 16:04:12.646449 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:12.646375    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-zbhlt" podStartSLOduration=2.422762132 podStartE2EDuration="3.6463626s" podCreationTimestamp="2026-05-26 16:04:09 +0000 UTC" firstStartedPulling="2026-05-26 16:04:10.365690116 +0000 UTC m=+167.789283255" lastFinishedPulling="2026-05-26 16:04:11.589290584 +0000 UTC m=+169.012883723" observedRunningTime="2026-05-26 16:04:12.645412127 +0000 UTC m=+170.069005282" watchObservedRunningTime="2026-05-26 16:04:12.6463626 +0000 UTC m=+170.069955757"
May 26 16:04:14.678152 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.678118    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"]
May 26 16:04:14.680215 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.680198    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"
May 26 16:04:14.682566 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.682539    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 26 16:04:14.682566 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.682553    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 26 16:04:14.682744 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.682557    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5m2hf\""
May 26 16:04:14.692322 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.692299    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"]
May 26 16:04:14.712911 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.712881    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fw8nm"]
May 26 16:04:14.715080 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.715066    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.717337 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.717317    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 26 16:04:14.717562 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.717538    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 26 16:04:14.717562 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.717548    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 26 16:04:14.717681 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.717587    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g5xd9\""
May 26 16:04:14.758338 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.758315    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43fccc63-8224-4fd6-8e3a-ef817181d328-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-mfjl9\" (UID: \"43fccc63-8224-4fd6-8e3a-ef817181d328\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"
May 26 16:04:14.758471 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.758351    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-wtmp\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.758471 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.758380    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-textfile\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.758590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.758467    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbsht\" (UniqueName: \"kubernetes.io/projected/0379f1a0-131d-447a-b24b-ad2c7e39bc50-kube-api-access-bbsht\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.758590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.758509    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-tls\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.758590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.758540    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43fccc63-8224-4fd6-8e3a-ef817181d328-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-mfjl9\" (UID: \"43fccc63-8224-4fd6-8e3a-ef817181d328\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"
May 26 16:04:14.758590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.758570    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43fccc63-8224-4fd6-8e3a-ef817181d328-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-mfjl9\" (UID: \"43fccc63-8224-4fd6-8e3a-ef817181d328\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"
May 26 16:04:14.758783 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.758632    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.758783 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.758666    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-accelerators-collector-config\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.758783 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.758716    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0379f1a0-131d-447a-b24b-ad2c7e39bc50-metrics-client-ca\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.758783 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.758749    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0379f1a0-131d-447a-b24b-ad2c7e39bc50-root\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.758783 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.758772    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmrh6\" (UniqueName: \"kubernetes.io/projected/43fccc63-8224-4fd6-8e3a-ef817181d328-kube-api-access-pmrh6\") pod \"openshift-state-metrics-9c959559f-mfjl9\" (UID: \"43fccc63-8224-4fd6-8e3a-ef817181d328\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"
May 26 16:04:14.759021 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.758800    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0379f1a0-131d-447a-b24b-ad2c7e39bc50-sys\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.859240 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.859210    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-wtmp\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.859240 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.859240    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-textfile\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.859446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.859259    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bbsht\" (UniqueName: \"kubernetes.io/projected/0379f1a0-131d-447a-b24b-ad2c7e39bc50-kube-api-access-bbsht\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.859446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.859275    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-tls\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.859446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.859298    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43fccc63-8224-4fd6-8e3a-ef817181d328-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-mfjl9\" (UID: \"43fccc63-8224-4fd6-8e3a-ef817181d328\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"
May 26 16:04:14.859446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.859324    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43fccc63-8224-4fd6-8e3a-ef817181d328-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-mfjl9\" (UID: \"43fccc63-8224-4fd6-8e3a-ef817181d328\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"
May 26 16:04:14.859446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.859351    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.859446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.859375    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-accelerators-collector-config\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.859446 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:04:14.859417    2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 26 16:04:14.859446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.859432    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0379f1a0-131d-447a-b24b-ad2c7e39bc50-metrics-client-ca\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.859889 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.859465    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0379f1a0-131d-447a-b24b-ad2c7e39bc50-root\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.859889 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.859490    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrh6\" (UniqueName: \"kubernetes.io/projected/43fccc63-8224-4fd6-8e3a-ef817181d328-kube-api-access-pmrh6\") pod \"openshift-state-metrics-9c959559f-mfjl9\" (UID: \"43fccc63-8224-4fd6-8e3a-ef817181d328\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"
May 26 16:04:14.859889 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.859517    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0379f1a0-131d-447a-b24b-ad2c7e39bc50-sys\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.859889 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.859557    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43fccc63-8224-4fd6-8e3a-ef817181d328-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-mfjl9\" (UID: \"43fccc63-8224-4fd6-8e3a-ef817181d328\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"
May 26 16:04:14.859889 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.859626    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-textfile\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.859889 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.859684    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0379f1a0-131d-447a-b24b-ad2c7e39bc50-root\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.859889 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.859757    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-wtmp\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.860213 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.860084    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0379f1a0-131d-447a-b24b-ad2c7e39bc50-metrics-client-ca\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.860271 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.860218    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43fccc63-8224-4fd6-8e3a-ef817181d328-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-mfjl9\" (UID: \"43fccc63-8224-4fd6-8e3a-ef817181d328\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"
May 26 16:04:14.860271 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:04:14.860257    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-tls podName:0379f1a0-131d-447a-b24b-ad2c7e39bc50 nodeName:}" failed. No retries permitted until 2026-05-26 16:04:15.360236297 +0000 UTC m=+172.783829443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-tls") pod "node-exporter-fw8nm" (UID: "0379f1a0-131d-447a-b24b-ad2c7e39bc50") : secret "node-exporter-tls" not found
May 26 16:04:14.860386 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:04:14.860284    2560 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
May 26 16:04:14.860386 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.860306    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0379f1a0-131d-447a-b24b-ad2c7e39bc50-sys\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.860386 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:04:14.860351    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43fccc63-8224-4fd6-8e3a-ef817181d328-openshift-state-metrics-tls podName:43fccc63-8224-4fd6-8e3a-ef817181d328 nodeName:}" failed. No retries permitted until 2026-05-26 16:04:15.360334011 +0000 UTC m=+172.783927150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/43fccc63-8224-4fd6-8e3a-ef817181d328-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-mfjl9" (UID: "43fccc63-8224-4fd6-8e3a-ef817181d328") : secret "openshift-state-metrics-tls" not found
May 26 16:04:14.860548 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.860512    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-accelerators-collector-config\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.862386 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.862348    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.862649 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.862629    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43fccc63-8224-4fd6-8e3a-ef817181d328-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-mfjl9\" (UID: \"43fccc63-8224-4fd6-8e3a-ef817181d328\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"
May 26 16:04:14.875614 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.875592    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbsht\" (UniqueName: \"kubernetes.io/projected/0379f1a0-131d-447a-b24b-ad2c7e39bc50-kube-api-access-bbsht\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:14.879356 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:14.879335    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrh6\" (UniqueName: \"kubernetes.io/projected/43fccc63-8224-4fd6-8e3a-ef817181d328-kube-api-access-pmrh6\") pod \"openshift-state-metrics-9c959559f-mfjl9\" (UID: \"43fccc63-8224-4fd6-8e3a-ef817181d328\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"
May 26 16:04:15.166268 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:15.166228    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:04:15.166477 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:15.166228    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:04:15.168762 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:15.168743    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-22s8h\""
May 26 16:04:15.177612 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:15.177591    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-29s87"
May 26 16:04:15.316735 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:15.316705    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-29s87"]
May 26 16:04:15.319242 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:04:15.319217    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcbf8b71_6b16_4832_b58b_537691709f83.slice/crio-67c608bfb73e6a8140c5d4b105dd1ea6568ddaa02a682d7cc9ea8a17ea50c019 WatchSource:0}: Error finding container 67c608bfb73e6a8140c5d4b105dd1ea6568ddaa02a682d7cc9ea8a17ea50c019: Status 404 returned error can't find the container with id 67c608bfb73e6a8140c5d4b105dd1ea6568ddaa02a682d7cc9ea8a17ea50c019
May 26 16:04:15.364052 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:15.364025    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-tls\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:15.364196 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:15.364103    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43fccc63-8224-4fd6-8e3a-ef817181d328-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-mfjl9\" (UID: \"43fccc63-8224-4fd6-8e3a-ef817181d328\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"
May 26 16:04:15.364243 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:04:15.364199    2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 26 16:04:15.364282 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:04:15.364265    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-tls podName:0379f1a0-131d-447a-b24b-ad2c7e39bc50 nodeName:}" failed. No retries permitted until 2026-05-26 16:04:16.364248374 +0000 UTC m=+173.787841521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-tls") pod "node-exporter-fw8nm" (UID: "0379f1a0-131d-447a-b24b-ad2c7e39bc50") : secret "node-exporter-tls" not found
May 26 16:04:15.366287 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:15.366262    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43fccc63-8224-4fd6-8e3a-ef817181d328-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-mfjl9\" (UID: \"43fccc63-8224-4fd6-8e3a-ef817181d328\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"
May 26 16:04:15.591767 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:15.591737    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"
May 26 16:04:15.642435 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:15.642257    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-29s87" event={"ID":"fcbf8b71-6b16-4832-b58b-537691709f83","Type":"ContainerStarted","Data":"67c608bfb73e6a8140c5d4b105dd1ea6568ddaa02a682d7cc9ea8a17ea50c019"}
May 26 16:04:15.732264 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:15.732209    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9"]
May 26 16:04:15.736453 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:04:15.736423    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43fccc63_8224_4fd6_8e3a_ef817181d328.slice/crio-33df87cfa8a9992b1abf6920c583ef13f4e27d27e65ea894ccb896c823796630 WatchSource:0}: Error finding container 33df87cfa8a9992b1abf6920c583ef13f4e27d27e65ea894ccb896c823796630: Status 404 returned error can't find the container with id 33df87cfa8a9992b1abf6920c583ef13f4e27d27e65ea894ccb896c823796630
May 26 16:04:16.375092 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.375057    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-tls\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:16.377167 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.377134    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0379f1a0-131d-447a-b24b-ad2c7e39bc50-node-exporter-tls\") pod \"node-exporter-fw8nm\" (UID: \"0379f1a0-131d-447a-b24b-ad2c7e39bc50\") " pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:16.523883 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.523850    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fw8nm"
May 26 16:04:16.649743 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.649651    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9" event={"ID":"43fccc63-8224-4fd6-8e3a-ef817181d328","Type":"ContainerStarted","Data":"df8924d958676e21a51e8248fd1ec8998a4e74e8e6873217afa4000e9058ce57"}
May 26 16:04:16.649743 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.649703    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9" event={"ID":"43fccc63-8224-4fd6-8e3a-ef817181d328","Type":"ContainerStarted","Data":"70371829e37b08ba2860b0c45adab8c8edb5675ad566b56daa7133388a0397a9"}
May 26 16:04:16.649743 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.649725    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9" event={"ID":"43fccc63-8224-4fd6-8e3a-ef817181d328","Type":"ContainerStarted","Data":"33df87cfa8a9992b1abf6920c583ef13f4e27d27e65ea894ccb896c823796630"}
May 26 16:04:16.781860 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.781819    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-66b5d866b4-7j58t"]
May 26 16:04:16.784558 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.784535    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.787171 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.787140    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5fosudm54lrcb\""
May 26 16:04:16.787171 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.787163    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 26 16:04:16.787350 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.787186    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 26 16:04:16.787350 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.787165    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 26 16:04:16.787350 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.787317    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 26 16:04:16.787350 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.787315    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7zhsf\""
May 26 16:04:16.787592 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.787536    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 26 16:04:16.796460 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.796440    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-66b5d866b4-7j58t"]
May 26 16:04:16.879166 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.879125    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.879166 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.879164    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlf8m\" (UniqueName: \"kubernetes.io/projected/a3dec2f0-7164-4ef4-a38b-9231faa02de5-kube-api-access-rlf8m\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.879414 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.879189    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.879414 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.879208    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3dec2f0-7164-4ef4-a38b-9231faa02de5-metrics-client-ca\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.879414 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.879277    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.879414 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.879364    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.879606 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.879457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-grpc-tls\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.879606 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.879504    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-thanos-querier-tls\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.886668 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:04:16.886643    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0379f1a0_131d_447a_b24b_ad2c7e39bc50.slice/crio-adec7542bcf83f3a590a41bfeddfb8b05d020629d17983ec537eeea8941a57c8 WatchSource:0}: Error finding container adec7542bcf83f3a590a41bfeddfb8b05d020629d17983ec537eeea8941a57c8: Status 404 returned error can't find the container with id adec7542bcf83f3a590a41bfeddfb8b05d020629d17983ec537eeea8941a57c8
May 26 16:04:16.980575 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.980542    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.980744 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.980598    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rlf8m\" (UniqueName: \"kubernetes.io/projected/a3dec2f0-7164-4ef4-a38b-9231faa02de5-kube-api-access-rlf8m\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.980744 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.980630    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.980744 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.980662    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3dec2f0-7164-4ef4-a38b-9231faa02de5-metrics-client-ca\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.980744 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.980703    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.980744 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.980738    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.981009 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.980894    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-grpc-tls\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.981009 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.980952    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-thanos-querier-tls\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.981492 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.981444    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3dec2f0-7164-4ef4-a38b-9231faa02de5-metrics-client-ca\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.983433 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.983410    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.983589 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.983566    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.984273 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.984226    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.984273 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.984265    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.984428 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.984280    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-thanos-querier-tls\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.984518 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.984498    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3dec2f0-7164-4ef4-a38b-9231faa02de5-secret-grpc-tls\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:16.988065 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:16.988039    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlf8m\" (UniqueName: \"kubernetes.io/projected/a3dec2f0-7164-4ef4-a38b-9231faa02de5-kube-api-access-rlf8m\") pod \"thanos-querier-66b5d866b4-7j58t\" (UID: \"a3dec2f0-7164-4ef4-a38b-9231faa02de5\") " pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:17.094855 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:17.094825    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:17.247728 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:17.247673    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-66b5d866b4-7j58t"]
May 26 16:04:17.252336 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:04:17.252307    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3dec2f0_7164_4ef4_a38b_9231faa02de5.slice/crio-da06b9f1690cb87b0c1ef994f4567f85a0cc5c2b68a0e519f3a48d2acf22e3ff WatchSource:0}: Error finding container da06b9f1690cb87b0c1ef994f4567f85a0cc5c2b68a0e519f3a48d2acf22e3ff: Status 404 returned error can't find the container with id da06b9f1690cb87b0c1ef994f4567f85a0cc5c2b68a0e519f3a48d2acf22e3ff
May 26 16:04:17.654834 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:17.654796    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fw8nm" event={"ID":"0379f1a0-131d-447a-b24b-ad2c7e39bc50","Type":"ContainerStarted","Data":"adec7542bcf83f3a590a41bfeddfb8b05d020629d17983ec537eeea8941a57c8"}
May 26 16:04:17.656260 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:17.656231    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-29s87" event={"ID":"fcbf8b71-6b16-4832-b58b-537691709f83","Type":"ContainerStarted","Data":"94d40f8eafe82660cb656cc9007fff10df47eeae8a6b223fb3ec0b7b688b0d17"}
May 26 16:04:17.658211 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:17.658178    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9" event={"ID":"43fccc63-8224-4fd6-8e3a-ef817181d328","Type":"ContainerStarted","Data":"8821150d663d3ea841ca1053a66ef97828f44834c02fed8cb791ba46b33fccdc"}
May 26 16:04:17.659358 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:17.659330    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t" event={"ID":"a3dec2f0-7164-4ef4-a38b-9231faa02de5","Type":"ContainerStarted","Data":"da06b9f1690cb87b0c1ef994f4567f85a0cc5c2b68a0e519f3a48d2acf22e3ff"}
May 26 16:04:17.672260 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:17.672213    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-29s87" podStartSLOduration=135.881727359 podStartE2EDuration="2m17.67219766s" podCreationTimestamp="2026-05-26 16:02:00 +0000 UTC" firstStartedPulling="2026-05-26 16:04:15.321209479 +0000 UTC m=+172.744802617" lastFinishedPulling="2026-05-26 16:04:17.111679765 +0000 UTC m=+174.535272918" observedRunningTime="2026-05-26 16:04:17.670652041 +0000 UTC m=+175.094245204" watchObservedRunningTime="2026-05-26 16:04:17.67219766 +0000 UTC m=+175.095790822"
May 26 16:04:17.688486 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:17.688444    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-mfjl9" podStartSLOduration=2.442022537 podStartE2EDuration="3.688432895s" podCreationTimestamp="2026-05-26 16:04:14 +0000 UTC" firstStartedPulling="2026-05-26 16:04:15.893536401 +0000 UTC m=+173.317129543" lastFinishedPulling="2026-05-26 16:04:17.139946748 +0000 UTC m=+174.563539901" observedRunningTime="2026-05-26 16:04:17.68694882 +0000 UTC m=+175.110541988" watchObservedRunningTime="2026-05-26 16:04:17.688432895 +0000 UTC m=+175.112026056"
May 26 16:04:18.663049 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.663016    2560 generic.go:358] "Generic (PLEG): container finished" podID="0379f1a0-131d-447a-b24b-ad2c7e39bc50" containerID="33647d9cfe1b41ba3e2f4ab76b63983789e28c4faedd2d65e8827945d67a75fe" exitCode=0
May 26 16:04:18.663434 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.663062    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fw8nm" event={"ID":"0379f1a0-131d-447a-b24b-ad2c7e39bc50","Type":"ContainerDied","Data":"33647d9cfe1b41ba3e2f4ab76b63983789e28c4faedd2d65e8827945d67a75fe"}
May 26 16:04:18.737868 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.737843    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ddcd7b94b-ddf28"]
May 26 16:04:18.739871 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.739850    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.742463 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.742348    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 26 16:04:18.742463 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.742368    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 26 16:04:18.742463 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.742409    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 26 16:04:18.742678 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.742539    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 26 16:04:18.742753 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.742731    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 26 16:04:18.742855 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.742801    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 26 16:04:18.742855 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.742828    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 26 16:04:18.742961 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.742856    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xfnzp\""
May 26 16:04:18.746872 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.746852    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 26 16:04:18.750637 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.750614    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ddcd7b94b-ddf28"]
May 26 16:04:18.797647 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.797619    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-service-ca\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.797792 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.797662    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-oauth-config\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.797792 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.797693    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-serving-cert\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.797908 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.797868    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-trusted-ca-bundle\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.797980 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.797959    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-oauth-serving-cert\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.798209 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.798059    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c96nz\" (UniqueName: \"kubernetes.io/projected/473da4b9-1d21-41fc-a3eb-f04e4802773f-kube-api-access-c96nz\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.798325 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.798307    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-config\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.898929 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.898890    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c96nz\" (UniqueName: \"kubernetes.io/projected/473da4b9-1d21-41fc-a3eb-f04e4802773f-kube-api-access-c96nz\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.899075 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.898957    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-config\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.899075 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.898996    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-service-ca\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.899075 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.899021    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-oauth-config\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.899217 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.899164    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-serving-cert\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.899270 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.899245    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-trusted-ca-bundle\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.899312 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.899300    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-oauth-serving-cert\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.899850 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.899816    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-service-ca\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.899850 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.899830    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-config\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.900014 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.899995    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-oauth-serving-cert\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.900408 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.900364    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-trusted-ca-bundle\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.901883 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.901857    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-serving-cert\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.902005 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.901984    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-oauth-config\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:18.907329 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:18.907306    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c96nz\" (UniqueName: \"kubernetes.io/projected/473da4b9-1d21-41fc-a3eb-f04e4802773f-kube-api-access-c96nz\") pod \"console-5ddcd7b94b-ddf28\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") " pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:19.052423 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:19.052372    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:19.377542 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:19.377511    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ddcd7b94b-ddf28"]
May 26 16:04:19.383495 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:04:19.383467    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473da4b9_1d21_41fc_a3eb_f04e4802773f.slice/crio-e567344d092ee8d58dbe8ec9d001318dba159b10ac1d1ecb594fcb82ca344826 WatchSource:0}: Error finding container e567344d092ee8d58dbe8ec9d001318dba159b10ac1d1ecb594fcb82ca344826: Status 404 returned error can't find the container with id e567344d092ee8d58dbe8ec9d001318dba159b10ac1d1ecb594fcb82ca344826
May 26 16:04:19.668039 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:19.667943    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t" event={"ID":"a3dec2f0-7164-4ef4-a38b-9231faa02de5","Type":"ContainerStarted","Data":"4c00e6e4b06ad370d3e6f231f30b7462f2131fa989070e6f27061e0c3e0b229e"}
May 26 16:04:19.668039 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:19.667988    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t" event={"ID":"a3dec2f0-7164-4ef4-a38b-9231faa02de5","Type":"ContainerStarted","Data":"421c831a010d6b41223f1ec712b2047cb926104e2524c92d61d75183f7b22a47"}
May 26 16:04:19.668039 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:19.668005    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t" event={"ID":"a3dec2f0-7164-4ef4-a38b-9231faa02de5","Type":"ContainerStarted","Data":"d43fee7b125593d371f1f2dd4a16863051dd6d318d1efad447e65c913b9bd7ca"}
May 26 16:04:19.669860 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:19.669836    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fw8nm" event={"ID":"0379f1a0-131d-447a-b24b-ad2c7e39bc50","Type":"ContainerStarted","Data":"d88d08bd8ef09c1fc71707f128381d6e4ded060d0f4e8a9986f52dd68f8c87c4"}
May 26 16:04:19.669983 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:19.669866    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fw8nm" event={"ID":"0379f1a0-131d-447a-b24b-ad2c7e39bc50","Type":"ContainerStarted","Data":"a09f51e46f272dfea6aa128ea591ad402a2a8076efb888fc3f6451a830ca0158"}
May 26 16:04:19.670910 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:19.670891    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ddcd7b94b-ddf28" event={"ID":"473da4b9-1d21-41fc-a3eb-f04e4802773f","Type":"ContainerStarted","Data":"e567344d092ee8d58dbe8ec9d001318dba159b10ac1d1ecb594fcb82ca344826"}
May 26 16:04:19.691008 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:19.690969    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fw8nm" podStartSLOduration=4.882069334 podStartE2EDuration="5.690957715s" podCreationTimestamp="2026-05-26 16:04:14 +0000 UTC" firstStartedPulling="2026-05-26 16:04:16.88821484 +0000 UTC m=+174.311807979" lastFinishedPulling="2026-05-26 16:04:17.697103205 +0000 UTC m=+175.120696360" observedRunningTime="2026-05-26 16:04:19.690483469 +0000 UTC m=+177.114076630" watchObservedRunningTime="2026-05-26 16:04:19.690957715 +0000 UTC m=+177.114550875"
May 26 16:04:20.349866 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.349821    2560 patch_prober.go:28] interesting pod/image-registry-86594ddd4-x9sdj container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 26 16:04:20.350046 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.349884    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-86594ddd4-x9sdj" podUID="19de7b34-20ec-4840-8901-96b1c50c1964" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:04:20.678705 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.678672    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t" event={"ID":"a3dec2f0-7164-4ef4-a38b-9231faa02de5","Type":"ContainerStarted","Data":"bf1810d05c37d3337b2abb8a4fb8e464a82128fb223bc08ef6ebd99eb6627738"}
May 26 16:04:20.679121 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.678715    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t" event={"ID":"a3dec2f0-7164-4ef4-a38b-9231faa02de5","Type":"ContainerStarted","Data":"e72f389f26cbbbd203fedc4f8b3e2df5811eb6827aa1191c38285640940b0afd"}
May 26 16:04:20.679121 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.678732    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t" event={"ID":"a3dec2f0-7164-4ef4-a38b-9231faa02de5","Type":"ContainerStarted","Data":"d3a86596fb2dfe7351a6844e8a031002b9714a40aae77fd7ab4c3b25ced91a69"}
May 26 16:04:20.705736 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.705137    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t" podStartSLOduration=1.504025885 podStartE2EDuration="4.705118421s" podCreationTimestamp="2026-05-26 16:04:16 +0000 UTC" firstStartedPulling="2026-05-26 16:04:17.2545049 +0000 UTC m=+174.678098053" lastFinishedPulling="2026-05-26 16:04:20.455597447 +0000 UTC m=+177.879190589" observedRunningTime="2026-05-26 16:04:20.702673146 +0000 UTC m=+178.126266307" watchObservedRunningTime="2026-05-26 16:04:20.705118421 +0000 UTC m=+178.128711585"
May 26 16:04:20.941717 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.940742    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:04:20.945693 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.945668    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:20.949956 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.949746    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 26 16:04:20.949956 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.949812    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 26 16:04:20.950471 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.950282    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 26 16:04:20.953239 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.953217    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 26 16:04:20.953430 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.953412    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 26 16:04:20.953514 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.953500    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 26 16:04:20.953634 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.953617    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a6t8trlaj41oc\""
May 26 16:04:20.954171 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.954152    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 26 16:04:20.955362 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.955343    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-s4jwn\""
May 26 16:04:20.957075 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.957051    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 26 16:04:20.957361 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.957057    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 26 16:04:20.957482 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.957137    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 26 16:04:20.958234 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.958213    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 26 16:04:20.959495 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.959477    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 26 16:04:20.969131 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.964063    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 26 16:04:20.973098 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:20.973079    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:04:21.020702 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.020670    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-web-config\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.020724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-config\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.020754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.020782    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ab1d95e-1514-4e62-8316-3d2730a48ca6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.020816    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.020840    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqbbj\" (UniqueName: \"kubernetes.io/projected/1ab1d95e-1514-4e62-8316-3d2730a48ca6-kube-api-access-xqbbj\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.020862    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.020921    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.020951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.021079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.021123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.021147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.021174    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.021216    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.021301    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.021333    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.021356    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.021446 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.021428    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ab1d95e-1514-4e62-8316-3d2730a48ca6-config-out\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.122326 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.122289    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.122537 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.122347    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.122537 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.122381    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.122537 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.122424    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.122537 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.122465    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.122537 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.122508    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.122537 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.122536    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.122836 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.122568    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.122836 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.122593    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ab1d95e-1514-4e62-8316-3d2730a48ca6-config-out\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.122836 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.122625    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-web-config\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.122836 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.122650    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-config\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.122836 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.122669    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.123436 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.123199    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.123436 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.123249    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.123982 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.123652    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.123982 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.123658    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ab1d95e-1514-4e62-8316-3d2730a48ca6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.123982 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.123701    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.123982 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.123727    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.123982 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.123760    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xqbbj\" (UniqueName: \"kubernetes.io/projected/1ab1d95e-1514-4e62-8316-3d2730a48ca6-kube-api-access-xqbbj\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.123982 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.123791    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.123982 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.123826    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.123982 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.123894    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.124950 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.124924    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.126455 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.126431    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.126865 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.126840    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.127076 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.127051    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ab1d95e-1514-4e62-8316-3d2730a48ca6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.127600 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.127556    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.129308 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.129285    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.130777 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.130273    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ab1d95e-1514-4e62-8316-3d2730a48ca6-config-out\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.130777 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.130342    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.130777 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.130483    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.130777 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.130485    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-web-config\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.130777 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.130713    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.131375 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.131337    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-config\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.131702 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.131665    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.135219 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.135201    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqbbj\" (UniqueName: \"kubernetes.io/projected/1ab1d95e-1514-4e62-8316-3d2730a48ca6-kube-api-access-xqbbj\") pod \"prometheus-k8s-0\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.262591 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.262200    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:21.423294 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.423237    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:04:21.425188 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:04:21.425152    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab1d95e_1514_4e62_8316_3d2730a48ca6.slice/crio-4b37a10d85d50a1ef6c405e3e47fc19848336bb4624ceeedce8f70abae78da25 WatchSource:0}: Error finding container 4b37a10d85d50a1ef6c405e3e47fc19848336bb4624ceeedce8f70abae78da25: Status 404 returned error can't find the container with id 4b37a10d85d50a1ef6c405e3e47fc19848336bb4624ceeedce8f70abae78da25
May 26 16:04:21.632933 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.632844    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v7sq4"
May 26 16:04:21.684194 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.684142    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ab1d95e-1514-4e62-8316-3d2730a48ca6","Type":"ContainerStarted","Data":"4b37a10d85d50a1ef6c405e3e47fc19848336bb4624ceeedce8f70abae78da25"}
May 26 16:04:21.684651 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:21.684509    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:22.689364 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:22.689335    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ddcd7b94b-ddf28" event={"ID":"473da4b9-1d21-41fc-a3eb-f04e4802773f","Type":"ContainerStarted","Data":"f08a06a6683da829b68e317fc7b34be0923f50de594426c10e60698d80c51224"}
May 26 16:04:22.707308 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:22.707263    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5ddcd7b94b-ddf28" podStartSLOduration=1.516873367 podStartE2EDuration="4.70724925s" podCreationTimestamp="2026-05-26 16:04:18 +0000 UTC" firstStartedPulling="2026-05-26 16:04:19.385444907 +0000 UTC m=+176.809038046" lastFinishedPulling="2026-05-26 16:04:22.575820787 +0000 UTC m=+179.999413929" observedRunningTime="2026-05-26 16:04:22.705257677 +0000 UTC m=+180.128850843" watchObservedRunningTime="2026-05-26 16:04:22.70724925 +0000 UTC m=+180.130842411"
May 26 16:04:23.693689 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:23.693654    2560 generic.go:358] "Generic (PLEG): container finished" podID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerID="b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a" exitCode=0
May 26 16:04:23.694119 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:23.693694    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ab1d95e-1514-4e62-8316-3d2730a48ca6","Type":"ContainerDied","Data":"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a"}
May 26 16:04:25.363476 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.363426    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-86594ddd4-x9sdj" podUID="19de7b34-20ec-4840-8901-96b1c50c1964" containerName="registry" containerID="cri-o://963b364b9ec6408d84bc39207aabf76de60154551140bf193c8c986a4f0f5358" gracePeriod=30
May 26 16:04:25.628869 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.628843    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:04:25.713438 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.713366    2560 generic.go:358] "Generic (PLEG): container finished" podID="19de7b34-20ec-4840-8901-96b1c50c1964" containerID="963b364b9ec6408d84bc39207aabf76de60154551140bf193c8c986a4f0f5358" exitCode=0
May 26 16:04:25.713438 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.713435    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86594ddd4-x9sdj" event={"ID":"19de7b34-20ec-4840-8901-96b1c50c1964","Type":"ContainerDied","Data":"963b364b9ec6408d84bc39207aabf76de60154551140bf193c8c986a4f0f5358"}
May 26 16:04:25.713664 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.713476    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86594ddd4-x9sdj"
May 26 16:04:25.713664 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.713491    2560 scope.go:117] "RemoveContainer" containerID="963b364b9ec6408d84bc39207aabf76de60154551140bf193c8c986a4f0f5358"
May 26 16:04:25.713664 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.713480    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86594ddd4-x9sdj" event={"ID":"19de7b34-20ec-4840-8901-96b1c50c1964","Type":"ContainerDied","Data":"729d75bd6b4fe87821048edf67eaf11beb71c5218b6df180624dbc5445c76af4"}
May 26 16:04:25.722328 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.722307    2560 scope.go:117] "RemoveContainer" containerID="963b364b9ec6408d84bc39207aabf76de60154551140bf193c8c986a4f0f5358"
May 26 16:04:25.722637 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:04:25.722615    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963b364b9ec6408d84bc39207aabf76de60154551140bf193c8c986a4f0f5358\": container with ID starting with 963b364b9ec6408d84bc39207aabf76de60154551140bf193c8c986a4f0f5358 not found: ID does not exist" containerID="963b364b9ec6408d84bc39207aabf76de60154551140bf193c8c986a4f0f5358"
May 26 16:04:25.722716 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.722653    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963b364b9ec6408d84bc39207aabf76de60154551140bf193c8c986a4f0f5358"} err="failed to get container status \"963b364b9ec6408d84bc39207aabf76de60154551140bf193c8c986a4f0f5358\": rpc error: code = NotFound desc = could not find container \"963b364b9ec6408d84bc39207aabf76de60154551140bf193c8c986a4f0f5358\": container with ID starting with 963b364b9ec6408d84bc39207aabf76de60154551140bf193c8c986a4f0f5358 not found: ID does not exist"
May 26 16:04:25.772938 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.772913    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19de7b34-20ec-4840-8901-96b1c50c1964-ca-trust-extracted\") pod \"19de7b34-20ec-4840-8901-96b1c50c1964\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") "
May 26 16:04:25.773076 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.772962    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19de7b34-20ec-4840-8901-96b1c50c1964-installation-pull-secrets\") pod \"19de7b34-20ec-4840-8901-96b1c50c1964\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") "
May 26 16:04:25.773076 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.772987    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gk92\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-kube-api-access-9gk92\") pod \"19de7b34-20ec-4840-8901-96b1c50c1964\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") "
May 26 16:04:25.773076 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.773016    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls\") pod \"19de7b34-20ec-4840-8901-96b1c50c1964\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") "
May 26 16:04:25.773076 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.773048    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19de7b34-20ec-4840-8901-96b1c50c1964-registry-certificates\") pod \"19de7b34-20ec-4840-8901-96b1c50c1964\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") "
May 26 16:04:25.773292 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.773085    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19de7b34-20ec-4840-8901-96b1c50c1964-trusted-ca\") pod \"19de7b34-20ec-4840-8901-96b1c50c1964\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") "
May 26 16:04:25.773292 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.773138    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19de7b34-20ec-4840-8901-96b1c50c1964-image-registry-private-configuration\") pod \"19de7b34-20ec-4840-8901-96b1c50c1964\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") "
May 26 16:04:25.773292 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.773221    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-bound-sa-token\") pod \"19de7b34-20ec-4840-8901-96b1c50c1964\" (UID: \"19de7b34-20ec-4840-8901-96b1c50c1964\") "
May 26 16:04:25.773671 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.773571    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19de7b34-20ec-4840-8901-96b1c50c1964-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "19de7b34-20ec-4840-8901-96b1c50c1964" (UID: "19de7b34-20ec-4840-8901-96b1c50c1964"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:25.773980 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.773952    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19de7b34-20ec-4840-8901-96b1c50c1964-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "19de7b34-20ec-4840-8901-96b1c50c1964" (UID: "19de7b34-20ec-4840-8901-96b1c50c1964"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:25.776003 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.775936    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "19de7b34-20ec-4840-8901-96b1c50c1964" (UID: "19de7b34-20ec-4840-8901-96b1c50c1964"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:04:25.776003 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.775979    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "19de7b34-20ec-4840-8901-96b1c50c1964" (UID: "19de7b34-20ec-4840-8901-96b1c50c1964"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:04:25.776003 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.776266    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-kube-api-access-9gk92" (OuterVolumeSpecName: "kube-api-access-9gk92") pod "19de7b34-20ec-4840-8901-96b1c50c1964" (UID: "19de7b34-20ec-4840-8901-96b1c50c1964"). InnerVolumeSpecName "kube-api-access-9gk92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:04:25.776003 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.776347    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19de7b34-20ec-4840-8901-96b1c50c1964-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "19de7b34-20ec-4840-8901-96b1c50c1964" (UID: "19de7b34-20ec-4840-8901-96b1c50c1964"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:25.778000 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.777973    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19de7b34-20ec-4840-8901-96b1c50c1964-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "19de7b34-20ec-4840-8901-96b1c50c1964" (UID: "19de7b34-20ec-4840-8901-96b1c50c1964"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:25.783551 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.783528    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19de7b34-20ec-4840-8901-96b1c50c1964-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "19de7b34-20ec-4840-8901-96b1c50c1964" (UID: "19de7b34-20ec-4840-8901-96b1c50c1964"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:04:25.873916 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.873857    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-bound-sa-token\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:04:25.873916 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.873884    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19de7b34-20ec-4840-8901-96b1c50c1964-ca-trust-extracted\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:04:25.873916 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.873894    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19de7b34-20ec-4840-8901-96b1c50c1964-installation-pull-secrets\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:04:25.873916 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.873904    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gk92\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-kube-api-access-9gk92\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:04:25.873916 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.873914    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19de7b34-20ec-4840-8901-96b1c50c1964-registry-tls\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:04:25.874140 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.873923    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19de7b34-20ec-4840-8901-96b1c50c1964-registry-certificates\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:04:25.874140 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.873932    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19de7b34-20ec-4840-8901-96b1c50c1964-trusted-ca\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:04:25.874140 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:25.873942    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19de7b34-20ec-4840-8901-96b1c50c1964-image-registry-private-configuration\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:04:26.041828 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:26.041793    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86594ddd4-x9sdj"]
May 26 16:04:26.045890 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:26.045864    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-86594ddd4-x9sdj"]
May 26 16:04:27.172242 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:27.172211    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19de7b34-20ec-4840-8901-96b1c50c1964" path="/var/lib/kubelet/pods/19de7b34-20ec-4840-8901-96b1c50c1964/volumes"
May 26 16:04:27.696042 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:27.696009    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-66b5d866b4-7j58t"
May 26 16:04:27.723580 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:27.723550    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ab1d95e-1514-4e62-8316-3d2730a48ca6","Type":"ContainerStarted","Data":"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b"}
May 26 16:04:27.723580 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:27.723580    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ab1d95e-1514-4e62-8316-3d2730a48ca6","Type":"ContainerStarted","Data":"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af"}
May 26 16:04:27.723753 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:27.723590    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ab1d95e-1514-4e62-8316-3d2730a48ca6","Type":"ContainerStarted","Data":"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69"}
May 26 16:04:27.723753 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:27.723598    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ab1d95e-1514-4e62-8316-3d2730a48ca6","Type":"ContainerStarted","Data":"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440"}
May 26 16:04:27.723753 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:27.723607    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ab1d95e-1514-4e62-8316-3d2730a48ca6","Type":"ContainerStarted","Data":"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9"}
May 26 16:04:27.723753 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:27.723615    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ab1d95e-1514-4e62-8316-3d2730a48ca6","Type":"ContainerStarted","Data":"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9"}
May 26 16:04:27.749087 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:27.749024    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.345129045 podStartE2EDuration="7.749006788s" podCreationTimestamp="2026-05-26 16:04:20 +0000 UTC" firstStartedPulling="2026-05-26 16:04:21.427194168 +0000 UTC m=+178.850787308" lastFinishedPulling="2026-05-26 16:04:26.831071908 +0000 UTC m=+184.254665051" observedRunningTime="2026-05-26 16:04:27.748073859 +0000 UTC m=+185.171667024" watchObservedRunningTime="2026-05-26 16:04:27.749006788 +0000 UTC m=+185.172599951"
May 26 16:04:29.052869 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:29.052836    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:29.053295 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:29.052879    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:29.057545 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:29.057525    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:29.732918 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:29.732886    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:04:31.263516 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:31.263481    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:41.034841 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:04:41.034808    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ddcd7b94b-ddf28"]
May 26 16:05:01.961168 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:01.961140    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v7sq4_b7627daa-7d0e-4a02-9b00-a3515e6b7137/dns/0.log"
May 26 16:05:01.967224 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:01.967208    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v7sq4_b7627daa-7d0e-4a02-9b00-a3515e6b7137/kube-rbac-proxy/0.log"
May 26 16:05:02.446726 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:02.446697    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xx9vq_a1e36e32-7353-4703-8b16-848a49a47e91/dns-node-resolver/0.log"
May 26 16:05:05.831714 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:05.831679    2560 generic.go:358] "Generic (PLEG): container finished" podID="8f4f1c38-12f6-4b6c-9360-0ba547d5f28a" containerID="18489eee4475ce4bf257cf2c98443bbb8588039e8b4b96231db858282bb20a90" exitCode=0
May 26 16:05:05.832075 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:05.831755    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t" event={"ID":"8f4f1c38-12f6-4b6c-9360-0ba547d5f28a","Type":"ContainerDied","Data":"18489eee4475ce4bf257cf2c98443bbb8588039e8b4b96231db858282bb20a90"}
May 26 16:05:05.832075 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:05.832062    2560 scope.go:117] "RemoveContainer" containerID="18489eee4475ce4bf257cf2c98443bbb8588039e8b4b96231db858282bb20a90"
May 26 16:05:06.054261 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.054221    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5ddcd7b94b-ddf28" podUID="473da4b9-1d21-41fc-a3eb-f04e4802773f" containerName="console" containerID="cri-o://f08a06a6683da829b68e317fc7b34be0923f50de594426c10e60698d80c51224" gracePeriod=15
May 26 16:05:06.299763 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.299742    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5ddcd7b94b-ddf28_473da4b9-1d21-41fc-a3eb-f04e4802773f/console/0.log"
May 26 16:05:06.299881 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.299801    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:05:06.425062 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.424973    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-oauth-config\") pod \"473da4b9-1d21-41fc-a3eb-f04e4802773f\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") "
May 26 16:05:06.425062 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.425012    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-trusted-ca-bundle\") pod \"473da4b9-1d21-41fc-a3eb-f04e4802773f\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") "
May 26 16:05:06.425062 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.425057    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-serving-cert\") pod \"473da4b9-1d21-41fc-a3eb-f04e4802773f\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") "
May 26 16:05:06.425326 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.425073    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-config\") pod \"473da4b9-1d21-41fc-a3eb-f04e4802773f\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") "
May 26 16:05:06.425326 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.425109    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-service-ca\") pod \"473da4b9-1d21-41fc-a3eb-f04e4802773f\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") "
May 26 16:05:06.425326 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.425126    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-oauth-serving-cert\") pod \"473da4b9-1d21-41fc-a3eb-f04e4802773f\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") "
May 26 16:05:06.425326 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.425191    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c96nz\" (UniqueName: \"kubernetes.io/projected/473da4b9-1d21-41fc-a3eb-f04e4802773f-kube-api-access-c96nz\") pod \"473da4b9-1d21-41fc-a3eb-f04e4802773f\" (UID: \"473da4b9-1d21-41fc-a3eb-f04e4802773f\") "
May 26 16:05:06.425570 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.425534    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "473da4b9-1d21-41fc-a3eb-f04e4802773f" (UID: "473da4b9-1d21-41fc-a3eb-f04e4802773f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:06.425684 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.425655    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "473da4b9-1d21-41fc-a3eb-f04e4802773f" (UID: "473da4b9-1d21-41fc-a3eb-f04e4802773f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:06.425684 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.425665    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-config" (OuterVolumeSpecName: "console-config") pod "473da4b9-1d21-41fc-a3eb-f04e4802773f" (UID: "473da4b9-1d21-41fc-a3eb-f04e4802773f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:06.425871 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.425793    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-service-ca" (OuterVolumeSpecName: "service-ca") pod "473da4b9-1d21-41fc-a3eb-f04e4802773f" (UID: "473da4b9-1d21-41fc-a3eb-f04e4802773f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:06.427407 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.427369    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "473da4b9-1d21-41fc-a3eb-f04e4802773f" (UID: "473da4b9-1d21-41fc-a3eb-f04e4802773f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:06.427491 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.427443    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473da4b9-1d21-41fc-a3eb-f04e4802773f-kube-api-access-c96nz" (OuterVolumeSpecName: "kube-api-access-c96nz") pod "473da4b9-1d21-41fc-a3eb-f04e4802773f" (UID: "473da4b9-1d21-41fc-a3eb-f04e4802773f"). InnerVolumeSpecName "kube-api-access-c96nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:05:06.427491 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.427451    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "473da4b9-1d21-41fc-a3eb-f04e4802773f" (UID: "473da4b9-1d21-41fc-a3eb-f04e4802773f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:06.526097 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.526059    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-service-ca\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:06.526097 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.526090    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-oauth-serving-cert\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:06.526097 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.526100    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c96nz\" (UniqueName: \"kubernetes.io/projected/473da4b9-1d21-41fc-a3eb-f04e4802773f-kube-api-access-c96nz\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:06.526316 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.526111    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-oauth-config\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:06.526316 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.526122    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-trusted-ca-bundle\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:06.526316 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.526132    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-serving-cert\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:06.526316 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.526141    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/473da4b9-1d21-41fc-a3eb-f04e4802773f-console-config\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:06.836827 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.836795    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mmd9t" event={"ID":"8f4f1c38-12f6-4b6c-9360-0ba547d5f28a","Type":"ContainerStarted","Data":"e4bf509855006b903631856df9e80bcaf8e812a0fadde2526c894d50c4feb0e6"}
May 26 16:05:06.837785 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.837770    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5ddcd7b94b-ddf28_473da4b9-1d21-41fc-a3eb-f04e4802773f/console/0.log"
May 26 16:05:06.837838 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.837806    2560 generic.go:358] "Generic (PLEG): container finished" podID="473da4b9-1d21-41fc-a3eb-f04e4802773f" containerID="f08a06a6683da829b68e317fc7b34be0923f50de594426c10e60698d80c51224" exitCode=2
May 26 16:05:06.837873 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.837848    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ddcd7b94b-ddf28" event={"ID":"473da4b9-1d21-41fc-a3eb-f04e4802773f","Type":"ContainerDied","Data":"f08a06a6683da829b68e317fc7b34be0923f50de594426c10e60698d80c51224"}
May 26 16:05:06.837873 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.837862    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ddcd7b94b-ddf28"
May 26 16:05:06.837942 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.837872    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ddcd7b94b-ddf28" event={"ID":"473da4b9-1d21-41fc-a3eb-f04e4802773f","Type":"ContainerDied","Data":"e567344d092ee8d58dbe8ec9d001318dba159b10ac1d1ecb594fcb82ca344826"}
May 26 16:05:06.837942 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.837893    2560 scope.go:117] "RemoveContainer" containerID="f08a06a6683da829b68e317fc7b34be0923f50de594426c10e60698d80c51224"
May 26 16:05:06.845912 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.845893    2560 scope.go:117] "RemoveContainer" containerID="f08a06a6683da829b68e317fc7b34be0923f50de594426c10e60698d80c51224"
May 26 16:05:06.846159 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:05:06.846139    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08a06a6683da829b68e317fc7b34be0923f50de594426c10e60698d80c51224\": container with ID starting with f08a06a6683da829b68e317fc7b34be0923f50de594426c10e60698d80c51224 not found: ID does not exist" containerID="f08a06a6683da829b68e317fc7b34be0923f50de594426c10e60698d80c51224"
May 26 16:05:06.846198 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.846168    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08a06a6683da829b68e317fc7b34be0923f50de594426c10e60698d80c51224"} err="failed to get container status \"f08a06a6683da829b68e317fc7b34be0923f50de594426c10e60698d80c51224\": rpc error: code = NotFound desc = could not find container \"f08a06a6683da829b68e317fc7b34be0923f50de594426c10e60698d80c51224\": container with ID starting with f08a06a6683da829b68e317fc7b34be0923f50de594426c10e60698d80c51224 not found: ID does not exist"
May 26 16:05:06.867625 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.867597    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ddcd7b94b-ddf28"]
May 26 16:05:06.875293 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:06.875269    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5ddcd7b94b-ddf28"]
May 26 16:05:07.170198 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:07.170119    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="473da4b9-1d21-41fc-a3eb-f04e4802773f" path="/var/lib/kubelet/pods/473da4b9-1d21-41fc-a3eb-f04e4802773f/volumes"
May 26 16:05:21.263012 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:21.262977    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:21.282035 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:21.282012    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:21.894140 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:21.894105    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:34.955384 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:34.955300    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs\") pod \"network-metrics-daemon-j528d\" (UID: \"22219f09-c007-46a5-b425-df729845bf45\") " pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:05:34.957762 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:34.957733    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22219f09-c007-46a5-b425-df729845bf45-metrics-certs\") pod \"network-metrics-daemon-j528d\" (UID: \"22219f09-c007-46a5-b425-df729845bf45\") " pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:05:34.969736 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:34.969708    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8ddfl\""
May 26 16:05:34.977813 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:34.977784    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j528d"
May 26 16:05:35.111593 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:35.111563    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j528d"]
May 26 16:05:35.115032 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:05:35.114999    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22219f09_c007_46a5_b425_df729845bf45.slice/crio-c26001165cf794e0548d9e6e6d7c3bd3267461da1bd5c32c5d12a37fa4dfa41c WatchSource:0}: Error finding container c26001165cf794e0548d9e6e6d7c3bd3267461da1bd5c32c5d12a37fa4dfa41c: Status 404 returned error can't find the container with id c26001165cf794e0548d9e6e6d7c3bd3267461da1bd5c32c5d12a37fa4dfa41c
May 26 16:05:35.922525 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:35.922486    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j528d" event={"ID":"22219f09-c007-46a5-b425-df729845bf45","Type":"ContainerStarted","Data":"c26001165cf794e0548d9e6e6d7c3bd3267461da1bd5c32c5d12a37fa4dfa41c"}
May 26 16:05:36.927054 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:36.927022    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j528d" event={"ID":"22219f09-c007-46a5-b425-df729845bf45","Type":"ContainerStarted","Data":"f58feae1192db672672ac0b185afb171535522a2f1dbc32aeb02f9eaa019955d"}
May 26 16:05:36.927054 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:36.927058    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j528d" event={"ID":"22219f09-c007-46a5-b425-df729845bf45","Type":"ContainerStarted","Data":"589393df5130e1d6c23bbdc9e3bda6690953d865188bc3a239c03f236bf759f7"}
May 26 16:05:36.943571 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:36.943064    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-j528d" podStartSLOduration=252.946722629 podStartE2EDuration="4m13.943045807s" podCreationTimestamp="2026-05-26 16:01:23 +0000 UTC" firstStartedPulling="2026-05-26 16:05:35.116858712 +0000 UTC m=+252.540451852" lastFinishedPulling="2026-05-26 16:05:36.113181877 +0000 UTC m=+253.536775030" observedRunningTime="2026-05-26 16:05:36.941728878 +0000 UTC m=+254.365322040" watchObservedRunningTime="2026-05-26 16:05:36.943045807 +0000 UTC m=+254.366638969"
May 26 16:05:39.276506 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.276467    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:05:39.277443 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.277407    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="prometheus" containerID="cri-o://de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9" gracePeriod=600
May 26 16:05:39.277607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.277428    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="kube-rbac-proxy" containerID="cri-o://4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af" gracePeriod=600
May 26 16:05:39.277607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.277479    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="kube-rbac-proxy-web" containerID="cri-o://a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69" gracePeriod=600
May 26 16:05:39.277607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.277424    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="thanos-sidecar" containerID="cri-o://e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440" gracePeriod=600
May 26 16:05:39.277607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.277543    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="config-reloader" containerID="cri-o://191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9" gracePeriod=600
May 26 16:05:39.277607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.277570    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="kube-rbac-proxy-thanos" containerID="cri-o://902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b" gracePeriod=600
May 26 16:05:39.532824 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.532764    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:39.593315 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593279    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-kubelet-serving-ca-bundle\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.593517 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593320    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-kube-rbac-proxy\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.593517 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593360    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-config\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.593643 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593557    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqbbj\" (UniqueName: \"kubernetes.io/projected/1ab1d95e-1514-4e62-8316-3d2730a48ca6-kube-api-access-xqbbj\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.593643 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593601    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-metrics-client-certs\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.593643 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593636    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ab1d95e-1514-4e62-8316-3d2730a48ca6-tls-assets\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.593802 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593662    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-thanos-prometheus-http-client-file\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.593802 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593707    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-metrics-client-ca\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.593802 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593713    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:39.593802 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593736    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-web-config\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.593802 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593765    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-k8s-db\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.593802 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593793    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-trusted-ca-bundle\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.594081 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593820    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-grpc-tls\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.594081 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593863    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.594081 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593898    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ab1d95e-1514-4e62-8316-3d2730a48ca6-config-out\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.594081 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593922    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-tls\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.594081 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593969    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.594081 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.593995    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-k8s-rulefiles-0\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.594081 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.594036    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-serving-certs-ca-bundle\") pod \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\" (UID: \"1ab1d95e-1514-4e62-8316-3d2730a48ca6\") "
May 26 16:05:39.594500 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.594236    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:39.594500 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.594248    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.594826 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.594798    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:39.595005 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.594977    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:39.595568 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.595540    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:05:39.596775 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.596585    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:39.597075 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.597026    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:39.597276 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.597238    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:39.598035 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.598007    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-config" (OuterVolumeSpecName: "config") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:39.598511 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.598371    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:39.598689 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.598661    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab1d95e-1514-4e62-8316-3d2730a48ca6-kube-api-access-xqbbj" (OuterVolumeSpecName: "kube-api-access-xqbbj") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "kube-api-access-xqbbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:05:39.598782 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.598719    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:39.599188 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.599154    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:39.599722 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.599694    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:39.599813 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.599707    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:39.600094 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.600075    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab1d95e-1514-4e62-8316-3d2730a48ca6-config-out" (OuterVolumeSpecName: "config-out") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:05:39.600261 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.600243    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab1d95e-1514-4e62-8316-3d2730a48ca6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:05:39.608633 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.608607    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-web-config" (OuterVolumeSpecName: "web-config") pod "1ab1d95e-1514-4e62-8316-3d2730a48ca6" (UID: "1ab1d95e-1514-4e62-8316-3d2730a48ca6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:39.695410 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695355    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.695410 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695407    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.695410 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695419    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.695626 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695429    2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-kube-rbac-proxy\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.695626 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695439    2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-config\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.695626 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695449    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqbbj\" (UniqueName: \"kubernetes.io/projected/1ab1d95e-1514-4e62-8316-3d2730a48ca6-kube-api-access-xqbbj\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.695626 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695458    2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-metrics-client-certs\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.695626 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695468    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ab1d95e-1514-4e62-8316-3d2730a48ca6-tls-assets\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.695626 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695477    2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.695626 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695485    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-configmap-metrics-client-ca\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.695626 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695495    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-web-config\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.695626 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695503    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-k8s-db\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.695626 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695511    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab1d95e-1514-4e62-8316-3d2730a48ca6-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.695626 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695519    2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-grpc-tls\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.695626 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695529    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.695626 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695537    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ab1d95e-1514-4e62-8316-3d2730a48ca6-config-out\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.695626 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.695547    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1ab1d95e-1514-4e62-8316-3d2730a48ca6-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:05:39.937991 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.937912    2560 generic.go:358] "Generic (PLEG): container finished" podID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerID="902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b" exitCode=0
May 26 16:05:39.937991 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.937937    2560 generic.go:358] "Generic (PLEG): container finished" podID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerID="4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af" exitCode=0
May 26 16:05:39.937991 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.937944    2560 generic.go:358] "Generic (PLEG): container finished" podID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerID="a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69" exitCode=0
May 26 16:05:39.937991 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.937949    2560 generic.go:358] "Generic (PLEG): container finished" podID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerID="e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440" exitCode=0
May 26 16:05:39.937991 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.937954    2560 generic.go:358] "Generic (PLEG): container finished" podID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerID="191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9" exitCode=0
May 26 16:05:39.937991 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.937960    2560 generic.go:358] "Generic (PLEG): container finished" podID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerID="de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9" exitCode=0
May 26 16:05:39.938333 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.938012    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:39.938333 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.938004    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ab1d95e-1514-4e62-8316-3d2730a48ca6","Type":"ContainerDied","Data":"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b"}
May 26 16:05:39.938333 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.938131    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ab1d95e-1514-4e62-8316-3d2730a48ca6","Type":"ContainerDied","Data":"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af"}
May 26 16:05:39.938333 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.938147    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ab1d95e-1514-4e62-8316-3d2730a48ca6","Type":"ContainerDied","Data":"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69"}
May 26 16:05:39.938333 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.938161    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ab1d95e-1514-4e62-8316-3d2730a48ca6","Type":"ContainerDied","Data":"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440"}
May 26 16:05:39.938333 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.938175    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ab1d95e-1514-4e62-8316-3d2730a48ca6","Type":"ContainerDied","Data":"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9"}
May 26 16:05:39.938333 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.938191    2560 scope.go:117] "RemoveContainer" containerID="902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b"
May 26 16:05:39.938333 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.938189    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ab1d95e-1514-4e62-8316-3d2730a48ca6","Type":"ContainerDied","Data":"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9"}
May 26 16:05:39.938333 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.938276    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ab1d95e-1514-4e62-8316-3d2730a48ca6","Type":"ContainerDied","Data":"4b37a10d85d50a1ef6c405e3e47fc19848336bb4624ceeedce8f70abae78da25"}
May 26 16:05:39.946371 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.946208    2560 scope.go:117] "RemoveContainer" containerID="4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af"
May 26 16:05:39.952953 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.952934    2560 scope.go:117] "RemoveContainer" containerID="a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69"
May 26 16:05:39.958954 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.958939    2560 scope.go:117] "RemoveContainer" containerID="e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440"
May 26 16:05:39.962092 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.962073    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:05:39.966491 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.966469    2560 scope.go:117] "RemoveContainer" containerID="191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9"
May 26 16:05:39.966899 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.966882    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:05:39.972676 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.972661    2560 scope.go:117] "RemoveContainer" containerID="de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9"
May 26 16:05:39.979403 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.979376    2560 scope.go:117] "RemoveContainer" containerID="b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a"
May 26 16:05:39.985318 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.985289    2560 scope.go:117] "RemoveContainer" containerID="902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b"
May 26 16:05:39.985571 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:05:39.985551    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b\": container with ID starting with 902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b not found: ID does not exist" containerID="902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b"
May 26 16:05:39.985654 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.985581    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b"} err="failed to get container status \"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b\": rpc error: code = NotFound desc = could not find container \"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b\": container with ID starting with 902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b not found: ID does not exist"
May 26 16:05:39.985654 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.985607    2560 scope.go:117] "RemoveContainer" containerID="4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af"
May 26 16:05:39.985836 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:05:39.985820    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af\": container with ID starting with 4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af not found: ID does not exist" containerID="4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af"
May 26 16:05:39.985902 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.985842    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af"} err="failed to get container status \"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af\": rpc error: code = NotFound desc = could not find container \"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af\": container with ID starting with 4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af not found: ID does not exist"
May 26 16:05:39.985902 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.985864    2560 scope.go:117] "RemoveContainer" containerID="a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69"
May 26 16:05:39.986133 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:05:39.986111    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69\": container with ID starting with a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69 not found: ID does not exist" containerID="a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69"
May 26 16:05:39.986170 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.986140    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69"} err="failed to get container status \"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69\": rpc error: code = NotFound desc = could not find container \"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69\": container with ID starting with a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69 not found: ID does not exist"
May 26 16:05:39.986170 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.986155    2560 scope.go:117] "RemoveContainer" containerID="e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440"
May 26 16:05:39.986352 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:05:39.986336    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440\": container with ID starting with e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440 not found: ID does not exist" containerID="e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440"
May 26 16:05:39.986499 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.986358    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440"} err="failed to get container status \"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440\": rpc error: code = NotFound desc = could not find container \"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440\": container with ID starting with e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440 not found: ID does not exist"
May 26 16:05:39.986499 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.986374    2560 scope.go:117] "RemoveContainer" containerID="191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9"
May 26 16:05:39.986660 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:05:39.986646    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9\": container with ID starting with 191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9 not found: ID does not exist" containerID="191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9"
May 26 16:05:39.986692 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.986664    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9"} err="failed to get container status \"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9\": rpc error: code = NotFound desc = could not find container \"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9\": container with ID starting with 191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9 not found: ID does not exist"
May 26 16:05:39.986692 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.986676    2560 scope.go:117] "RemoveContainer" containerID="de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9"
May 26 16:05:39.986893 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:05:39.986876    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9\": container with ID starting with de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9 not found: ID does not exist" containerID="de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9"
May 26 16:05:39.986936 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.986899    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9"} err="failed to get container status \"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9\": rpc error: code = NotFound desc = could not find container \"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9\": container with ID starting with de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9 not found: ID does not exist"
May 26 16:05:39.986936 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.986914    2560 scope.go:117] "RemoveContainer" containerID="b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a"
May 26 16:05:39.987145 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:05:39.987127    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a\": container with ID starting with b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a not found: ID does not exist" containerID="b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a"
May 26 16:05:39.987219 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.987151    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a"} err="failed to get container status \"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a\": rpc error: code = NotFound desc = could not find container \"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a\": container with ID starting with b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a not found: ID does not exist"
May 26 16:05:39.987219 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.987172    2560 scope.go:117] "RemoveContainer" containerID="902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b"
May 26 16:05:39.987381 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.987365    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b"} err="failed to get container status \"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b\": rpc error: code = NotFound desc = could not find container \"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b\": container with ID starting with 902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b not found: ID does not exist"
May 26 16:05:39.987476 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.987383    2560 scope.go:117] "RemoveContainer" containerID="4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af"
May 26 16:05:39.987625 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.987607    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af"} err="failed to get container status \"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af\": rpc error: code = NotFound desc = could not find container \"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af\": container with ID starting with 4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af not found: ID does not exist"
May 26 16:05:39.987673 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.987626    2560 scope.go:117] "RemoveContainer" containerID="a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69"
May 26 16:05:39.987838 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.987820    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69"} err="failed to get container status \"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69\": rpc error: code = NotFound desc = could not find container \"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69\": container with ID starting with a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69 not found: ID does not exist"
May 26 16:05:39.987901 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.987838    2560 scope.go:117] "RemoveContainer" containerID="e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440"
May 26 16:05:39.988030 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.988015    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440"} err="failed to get container status \"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440\": rpc error: code = NotFound desc = could not find container \"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440\": container with ID starting with e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440 not found: ID does not exist"
May 26 16:05:39.988075 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.988030    2560 scope.go:117] "RemoveContainer" containerID="191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9"
May 26 16:05:39.988209 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.988192    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9"} err="failed to get container status \"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9\": rpc error: code = NotFound desc = could not find container \"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9\": container with ID starting with 191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9 not found: ID does not exist"
May 26 16:05:39.988252 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.988210    2560 scope.go:117] "RemoveContainer" containerID="de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9"
May 26 16:05:39.988435 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.988418    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9"} err="failed to get container status \"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9\": rpc error: code = NotFound desc = could not find container \"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9\": container with ID starting with de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9 not found: ID does not exist"
May 26 16:05:39.988497 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.988437    2560 scope.go:117] "RemoveContainer" containerID="b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a"
May 26 16:05:39.988620 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.988605    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a"} err="failed to get container status \"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a\": rpc error: code = NotFound desc = could not find container \"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a\": container with ID starting with b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a not found: ID does not exist"
May 26 16:05:39.988680 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.988621    2560 scope.go:117] "RemoveContainer" containerID="902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b"
May 26 16:05:39.988810 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.988794    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b"} err="failed to get container status \"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b\": rpc error: code = NotFound desc = could not find container \"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b\": container with ID starting with 902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b not found: ID does not exist"
May 26 16:05:39.988860 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.988811    2560 scope.go:117] "RemoveContainer" containerID="4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af"
May 26 16:05:39.989016 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.988999    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af"} err="failed to get container status \"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af\": rpc error: code = NotFound desc = could not find container \"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af\": container with ID starting with 4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af not found: ID does not exist"
May 26 16:05:39.989075 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.989017    2560 scope.go:117] "RemoveContainer" containerID="a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69"
May 26 16:05:39.989203 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.989185    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69"} err="failed to get container status \"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69\": rpc error: code = NotFound desc = could not find container \"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69\": container with ID starting with a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69 not found: ID does not exist"
May 26 16:05:39.989245 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.989203    2560 scope.go:117] "RemoveContainer" containerID="e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440"
May 26 16:05:39.989424 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.989406    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440"} err="failed to get container status \"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440\": rpc error: code = NotFound desc = could not find container \"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440\": container with ID starting with e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440 not found: ID does not exist"
May 26 16:05:39.989474 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.989425    2560 scope.go:117] "RemoveContainer" containerID="191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9"
May 26 16:05:39.989644 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.989627    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9"} err="failed to get container status \"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9\": rpc error: code = NotFound desc = could not find container \"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9\": container with ID starting with 191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9 not found: ID does not exist"
May 26 16:05:39.989713 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.989645    2560 scope.go:117] "RemoveContainer" containerID="de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9"
May 26 16:05:39.989862 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.989845    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9"} err="failed to get container status \"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9\": rpc error: code = NotFound desc = could not find container \"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9\": container with ID starting with de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9 not found: ID does not exist"
May 26 16:05:39.989914 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.989862    2560 scope.go:117] "RemoveContainer" containerID="b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a"
May 26 16:05:39.990060 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.990045    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a"} err="failed to get container status \"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a\": rpc error: code = NotFound desc = could not find container \"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a\": container with ID starting with b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a not found: ID does not exist"
May 26 16:05:39.990102 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.990061    2560 scope.go:117] "RemoveContainer" containerID="902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b"
May 26 16:05:39.990273 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.990255    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b"} err="failed to get container status \"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b\": rpc error: code = NotFound desc = could not find container \"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b\": container with ID starting with 902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b not found: ID does not exist"
May 26 16:05:39.990333 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.990275    2560 scope.go:117] "RemoveContainer" containerID="4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af"
May 26 16:05:39.990613 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.990569    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af"} err="failed to get container status \"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af\": rpc error: code = NotFound desc = could not find container \"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af\": container with ID starting with 4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af not found: ID does not exist"
May 26 16:05:39.990797 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.990615    2560 scope.go:117] "RemoveContainer" containerID="a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69"
May 26 16:05:39.991018 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.990993    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69"} err="failed to get container status \"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69\": rpc error: code = NotFound desc = could not find container \"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69\": container with ID starting with a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69 not found: ID does not exist"
May 26 16:05:39.991018 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.991017    2560 scope.go:117] "RemoveContainer" containerID="e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440"
May 26 16:05:39.991298 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.991280    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440"} err="failed to get container status \"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440\": rpc error: code = NotFound desc = could not find container \"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440\": container with ID starting with e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440 not found: ID does not exist"
May 26 16:05:39.991364 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.991300    2560 scope.go:117] "RemoveContainer" containerID="191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9"
May 26 16:05:39.991583 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.991559    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9"} err="failed to get container status \"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9\": rpc error: code = NotFound desc = could not find container \"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9\": container with ID starting with 191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9 not found: ID does not exist"
May 26 16:05:39.991663 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.991584    2560 scope.go:117] "RemoveContainer" containerID="de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9"
May 26 16:05:39.991843 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.991789    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9"} err="failed to get container status \"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9\": rpc error: code = NotFound desc = could not find container \"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9\": container with ID starting with de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9 not found: ID does not exist"
May 26 16:05:39.991932 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.991844    2560 scope.go:117] "RemoveContainer" containerID="b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a"
May 26 16:05:39.992108 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.992082    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a"} err="failed to get container status \"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a\": rpc error: code = NotFound desc = could not find container \"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a\": container with ID starting with b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a not found: ID does not exist"
May 26 16:05:39.992159 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.992109    2560 scope.go:117] "RemoveContainer" containerID="902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b"
May 26 16:05:39.992684 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.992602    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b"} err="failed to get container status \"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b\": rpc error: code = NotFound desc = could not find container \"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b\": container with ID starting with 902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b not found: ID does not exist"
May 26 16:05:39.992684 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.992631    2560 scope.go:117] "RemoveContainer" containerID="4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af"
May 26 16:05:39.993126 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.993102    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af"} err="failed to get container status \"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af\": rpc error: code = NotFound desc = could not find container \"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af\": container with ID starting with 4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af not found: ID does not exist"
May 26 16:05:39.993223 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.993137    2560 scope.go:117] "RemoveContainer" containerID="a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69"
May 26 16:05:39.993422 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.993372    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69"} err="failed to get container status \"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69\": rpc error: code = NotFound desc = could not find container \"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69\": container with ID starting with a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69 not found: ID does not exist"
May 26 16:05:39.993506 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.993427    2560 scope.go:117] "RemoveContainer" containerID="e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440"
May 26 16:05:39.993506 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.993406    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:05:39.993716 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.993689    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440"} err="failed to get container status \"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440\": rpc error: code = NotFound desc = could not find container \"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440\": container with ID starting with e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440 not found: ID does not exist"
May 26 16:05:39.993776 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.993720    2560 scope.go:117] "RemoveContainer" containerID="191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9"
May 26 16:05:39.993988 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.993959    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9"} err="failed to get container status \"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9\": rpc error: code = NotFound desc = could not find container \"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9\": container with ID starting with 191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9 not found: ID does not exist"
May 26 16:05:39.994063 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.993992    2560 scope.go:117] "RemoveContainer" containerID="de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9"
May 26 16:05:39.994115 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994079    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="config-reloader"
May 26 16:05:39.994115 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994096    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="config-reloader"
May 26 16:05:39.994212 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994120    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="thanos-sidecar"
May 26 16:05:39.994212 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994129    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="thanos-sidecar"
May 26 16:05:39.994212 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994159    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="kube-rbac-proxy"
May 26 16:05:39.994212 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994169    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="kube-rbac-proxy"
May 26 16:05:39.994212 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994196    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19de7b34-20ec-4840-8901-96b1c50c1964" containerName="registry"
May 26 16:05:39.994212 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994205    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="19de7b34-20ec-4840-8901-96b1c50c1964" containerName="registry"
May 26 16:05:39.994524 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994232    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="prometheus"
May 26 16:05:39.994524 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994242    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="prometheus"
May 26 16:05:39.994524 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994258    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="473da4b9-1d21-41fc-a3eb-f04e4802773f" containerName="console"
May 26 16:05:39.994524 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994266    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="473da4b9-1d21-41fc-a3eb-f04e4802773f" containerName="console"
May 26 16:05:39.994524 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994285    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="kube-rbac-proxy-thanos"
May 26 16:05:39.994524 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994296    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="kube-rbac-proxy-thanos"
May 26 16:05:39.994524 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994307    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="kube-rbac-proxy-web"
May 26 16:05:39.994524 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994316    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="kube-rbac-proxy-web"
May 26 16:05:39.994524 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994337    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="init-config-reloader"
May 26 16:05:39.994524 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994347    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="init-config-reloader"
May 26 16:05:39.994524 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994507    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="config-reloader"
May 26 16:05:39.994524 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994520    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="kube-rbac-proxy"
May 26 16:05:39.995099 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994534    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="kube-rbac-proxy-web"
May 26 16:05:39.995099 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994546    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="kube-rbac-proxy-thanos"
May 26 16:05:39.995099 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994565    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="19de7b34-20ec-4840-8901-96b1c50c1964" containerName="registry"
May 26 16:05:39.995099 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994576    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="473da4b9-1d21-41fc-a3eb-f04e4802773f" containerName="console"
May 26 16:05:39.995099 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994593    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="prometheus"
May 26 16:05:39.995099 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.994605    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" containerName="thanos-sidecar"
May 26 16:05:39.997828 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.995850    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9"} err="failed to get container status \"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9\": rpc error: code = NotFound desc = could not find container \"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9\": container with ID starting with de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9 not found: ID does not exist"
May 26 16:05:39.997828 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.995876    2560 scope.go:117] "RemoveContainer" containerID="b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a"
May 26 16:05:39.998061 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.997903    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a"} err="failed to get container status \"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a\": rpc error: code = NotFound desc = could not find container \"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a\": container with ID starting with b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a not found: ID does not exist"
May 26 16:05:39.998061 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.997932    2560 scope.go:117] "RemoveContainer" containerID="902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b"
May 26 16:05:39.998965 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.998593    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b"} err="failed to get container status \"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b\": rpc error: code = NotFound desc = could not find container \"902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b\": container with ID starting with 902a7b2de349f16c083384745bd83905ba368e7ff19f89113e905cd930beef3b not found: ID does not exist"
May 26 16:05:39.999062 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.998972    2560 scope.go:117] "RemoveContainer" containerID="4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af"
May 26 16:05:39.999259 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.999232    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af"} err="failed to get container status \"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af\": rpc error: code = NotFound desc = could not find container \"4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af\": container with ID starting with 4b44dbe832a31ffd8de5064679152063b7ad582a0c665bb2b6b25b4aa8cb21af not found: ID does not exist"
May 26 16:05:39.999259 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.999254    2560 scope.go:117] "RemoveContainer" containerID="a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69"
May 26 16:05:39.999551 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.999532    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69"} err="failed to get container status \"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69\": rpc error: code = NotFound desc = could not find container \"a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69\": container with ID starting with a7de0ae1b41fe183bb46ffe08984ae857ae4a0ae6a14b85fb409b89546ee2f69 not found: ID does not exist"
May 26 16:05:39.999610 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.999552    2560 scope.go:117] "RemoveContainer" containerID="e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440"
May 26 16:05:39.999788 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.999766    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440"} err="failed to get container status \"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440\": rpc error: code = NotFound desc = could not find container \"e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440\": container with ID starting with e9fa2ab97f95e8c9ea42be6bf32f16de80cb6eb028deac5e46a81b7f18c60440 not found: ID does not exist"
May 26 16:05:39.999870 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:39.999789    2560 scope.go:117] "RemoveContainer" containerID="191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9"
May 26 16:05:40.000042 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.000025    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9"} err="failed to get container status \"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9\": rpc error: code = NotFound desc = could not find container \"191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9\": container with ID starting with 191087193e9eafc6a58d32b1321e08aabb76dd0d4942a8890644991d3224b8e9 not found: ID does not exist"
May 26 16:05:40.000110 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.000044    2560 scope.go:117] "RemoveContainer" containerID="de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9"
May 26 16:05:40.000282 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.000264    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9"} err="failed to get container status \"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9\": rpc error: code = NotFound desc = could not find container \"de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9\": container with ID starting with de0fc13b4eb3fe55469b9136d739f9afc1942f2d3935f7b5275565a11e7f52f9 not found: ID does not exist"
May 26 16:05:40.000333 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.000283    2560 scope.go:117] "RemoveContainer" containerID="b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a"
May 26 16:05:40.000514 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.000498    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a"} err="failed to get container status \"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a\": rpc error: code = NotFound desc = could not find container \"b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a\": container with ID starting with b71f56f504181ca3ea3761b4919ea61199fc061a6f24fd20d0f8ba03bb00476a not found: ID does not exist"
May 26 16:05:40.001706 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.001689    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.004581 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.004565    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a6t8trlaj41oc\""
May 26 16:05:40.004658 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.004564    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 26 16:05:40.004827 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.004812    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 26 16:05:40.005051 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.005038    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 26 16:05:40.005177 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.005163    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 26 16:05:40.010240 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.010221    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 26 16:05:40.010449 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.010432    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-s4jwn\""
May 26 16:05:40.010672 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.010655    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 26 16:05:40.010672 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.010673    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 26 16:05:40.010672 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.010679    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 26 16:05:40.011024 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.011002    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 26 16:05:40.011141 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.011100    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 26 16:05:40.011141 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.011137    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 26 16:05:40.012284 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.012265    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:05:40.013808 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.013791    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 26 16:05:40.023215 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.023190    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 26 16:05:40.099371 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099331    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.099371 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099374    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.099657 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099410    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.099657 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.099657 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.099657 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099558    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.099657 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099583    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.099657 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099632    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.099907 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.099907 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099706    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.099907 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.099907 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099759    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.099907 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099813    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.099907 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099840    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.099907 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099858    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngsgl\" (UniqueName: \"kubernetes.io/projected/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-kube-api-access-ngsgl\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.100165 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099907    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-config\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.100165 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.100165 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.099975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201197 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201106    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201197 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201152    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201197 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201171    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ngsgl\" (UniqueName: \"kubernetes.io/projected/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-kube-api-access-ngsgl\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201520 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201283    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-config\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201520 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201318    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201520 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201350    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201520 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201414    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201520 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201449    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201520 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201473    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201520 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201496    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201853 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201572    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201853 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201647    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201853 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201674    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201853 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201702    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201853 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201741    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201853 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201776    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201853 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201816    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.201853 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201839    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.202250 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.201873    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.202250 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.202020    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.202694 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.202668    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.202791 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.202706    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.203336 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.203311    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.205050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.204110    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.205050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.204320    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-config\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.205050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.204348    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.205050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.204700    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.205050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.204766    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.205659 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.205614    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.205981 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.205959    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.206654 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.206635    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.206654 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.206645    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.206766 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.206733    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.206800 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.206773    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.207412 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.207369    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.208771 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.208755    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngsgl\" (UniqueName: \"kubernetes.io/projected/3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2-kube-api-access-ngsgl\") pod \"prometheus-k8s-0\" (UID: \"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.311619 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.311586    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:40.442756 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.442721    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:05:40.446417 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:05:40.446368    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca69fa1_9ef6_4e4e_9018_bbc8e11f4ec2.slice/crio-3a91199da379bfb7908420f8031f6dc765ca43afa6a3673d77af7b306c36852a WatchSource:0}: Error finding container 3a91199da379bfb7908420f8031f6dc765ca43afa6a3673d77af7b306c36852a: Status 404 returned error can't find the container with id 3a91199da379bfb7908420f8031f6dc765ca43afa6a3673d77af7b306c36852a
May 26 16:05:40.942906 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.942872    2560 generic.go:358] "Generic (PLEG): container finished" podID="3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2" containerID="5c3e0699f4e6f152b6cadf4bf046c5d7550b6778f545e37030d9664b7b73093b" exitCode=0
May 26 16:05:40.943069 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.942957    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2","Type":"ContainerDied","Data":"5c3e0699f4e6f152b6cadf4bf046c5d7550b6778f545e37030d9664b7b73093b"}
May 26 16:05:40.943069 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:40.943004    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2","Type":"ContainerStarted","Data":"3a91199da379bfb7908420f8031f6dc765ca43afa6a3673d77af7b306c36852a"}
May 26 16:05:41.174971 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:41.174930    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ab1d95e-1514-4e62-8316-3d2730a48ca6" path="/var/lib/kubelet/pods/1ab1d95e-1514-4e62-8316-3d2730a48ca6/volumes"
May 26 16:05:41.949346 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:41.949308    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2","Type":"ContainerStarted","Data":"67a1a4459c5e253902d2bd1edc5e627bc3bb2ce1460131b48270c490d3d4101c"}
May 26 16:05:41.949346 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:41.949345    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2","Type":"ContainerStarted","Data":"4d4e3ce5ae427fae87bf569b1e6f9da1aee98882b7e5d8463cf8ddae5e86eac1"}
May 26 16:05:41.949346 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:41.949355    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2","Type":"ContainerStarted","Data":"fc9eed79254fa233c116757dce12330f6c7663a61f072d9273b5885590a6c2a2"}
May 26 16:05:41.949935 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:41.949364    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2","Type":"ContainerStarted","Data":"920456e6e4a387e9d778b7ce22baf1dea1e3e1c381fbb19294c043053de57a04"}
May 26 16:05:41.949935 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:41.949372    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2","Type":"ContainerStarted","Data":"baca6488d42706e03466de37ab9766cdc150120c603505057d2a8aef4ba3b9c9"}
May 26 16:05:41.949935 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:41.949380    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2","Type":"ContainerStarted","Data":"c3d914344f30dfea141fdc518ad02c900867a49ed8999861311526e52d425dd4"}
May 26 16:05:41.977348 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:41.977297    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.977283096 podStartE2EDuration="2.977283096s" podCreationTimestamp="2026-05-26 16:05:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:05:41.974820931 +0000 UTC m=+259.398414093" watchObservedRunningTime="2026-05-26 16:05:41.977283096 +0000 UTC m=+259.400876257"
May 26 16:05:45.312503 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:05:45.312464    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:06:23.028824 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:06:23.028797    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:06:23.029363 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:06:23.028856    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:06:23.038272 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:06:23.038246    2560 kubelet.go:1628] "Image garbage collection succeeded"
May 26 16:06:40.312437 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:06:40.312370    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:06:40.327652 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:06:40.327628    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:06:41.139950 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:06:41.139924    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:10:17.642225 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:17.642188    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-c7gdw"]
May 26 16:10:17.645868 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:17.645849    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7gdw"
May 26 16:10:17.648021 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:17.648000    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 16:10:17.653863 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:17.653841    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c7gdw"]
May 26 16:10:17.741825 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:17.741796    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/915dcfd8-ae6f-41be-9694-0f9eb624df9e-dbus\") pod \"global-pull-secret-syncer-c7gdw\" (UID: \"915dcfd8-ae6f-41be-9694-0f9eb624df9e\") " pod="kube-system/global-pull-secret-syncer-c7gdw"
May 26 16:10:17.741983 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:17.741828    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/915dcfd8-ae6f-41be-9694-0f9eb624df9e-kubelet-config\") pod \"global-pull-secret-syncer-c7gdw\" (UID: \"915dcfd8-ae6f-41be-9694-0f9eb624df9e\") " pod="kube-system/global-pull-secret-syncer-c7gdw"
May 26 16:10:17.741983 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:17.741926    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/915dcfd8-ae6f-41be-9694-0f9eb624df9e-original-pull-secret\") pod \"global-pull-secret-syncer-c7gdw\" (UID: \"915dcfd8-ae6f-41be-9694-0f9eb624df9e\") " pod="kube-system/global-pull-secret-syncer-c7gdw"
May 26 16:10:17.843252 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:17.843216    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/915dcfd8-ae6f-41be-9694-0f9eb624df9e-dbus\") pod \"global-pull-secret-syncer-c7gdw\" (UID: \"915dcfd8-ae6f-41be-9694-0f9eb624df9e\") " pod="kube-system/global-pull-secret-syncer-c7gdw"
May 26 16:10:17.843252 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:17.843254    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/915dcfd8-ae6f-41be-9694-0f9eb624df9e-kubelet-config\") pod \"global-pull-secret-syncer-c7gdw\" (UID: \"915dcfd8-ae6f-41be-9694-0f9eb624df9e\") " pod="kube-system/global-pull-secret-syncer-c7gdw"
May 26 16:10:17.843504 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:17.843298    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/915dcfd8-ae6f-41be-9694-0f9eb624df9e-original-pull-secret\") pod \"global-pull-secret-syncer-c7gdw\" (UID: \"915dcfd8-ae6f-41be-9694-0f9eb624df9e\") " pod="kube-system/global-pull-secret-syncer-c7gdw"
May 26 16:10:17.843504 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:17.843413    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/915dcfd8-ae6f-41be-9694-0f9eb624df9e-dbus\") pod \"global-pull-secret-syncer-c7gdw\" (UID: \"915dcfd8-ae6f-41be-9694-0f9eb624df9e\") " pod="kube-system/global-pull-secret-syncer-c7gdw"
May 26 16:10:17.843504 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:17.843413    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/915dcfd8-ae6f-41be-9694-0f9eb624df9e-kubelet-config\") pod \"global-pull-secret-syncer-c7gdw\" (UID: \"915dcfd8-ae6f-41be-9694-0f9eb624df9e\") " pod="kube-system/global-pull-secret-syncer-c7gdw"
May 26 16:10:17.845465 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:17.845450    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/915dcfd8-ae6f-41be-9694-0f9eb624df9e-original-pull-secret\") pod \"global-pull-secret-syncer-c7gdw\" (UID: \"915dcfd8-ae6f-41be-9694-0f9eb624df9e\") " pod="kube-system/global-pull-secret-syncer-c7gdw"
May 26 16:10:17.956082 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:17.955992    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7gdw"
May 26 16:10:18.076366 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:18.076336    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c7gdw"]
May 26 16:10:18.079469 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:10:18.079439    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915dcfd8_ae6f_41be_9694_0f9eb624df9e.slice/crio-99ff0ad19c0fd035698e10d7b10fd075353aa25adf88d2551d600fdad7a29293 WatchSource:0}: Error finding container 99ff0ad19c0fd035698e10d7b10fd075353aa25adf88d2551d600fdad7a29293: Status 404 returned error can't find the container with id 99ff0ad19c0fd035698e10d7b10fd075353aa25adf88d2551d600fdad7a29293
May 26 16:10:18.080975 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:18.080958    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:10:18.734222 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:18.734184    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c7gdw" event={"ID":"915dcfd8-ae6f-41be-9694-0f9eb624df9e","Type":"ContainerStarted","Data":"99ff0ad19c0fd035698e10d7b10fd075353aa25adf88d2551d600fdad7a29293"}
May 26 16:10:22.747411 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:22.747300    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c7gdw" event={"ID":"915dcfd8-ae6f-41be-9694-0f9eb624df9e","Type":"ContainerStarted","Data":"d0aa4deee749653d4a425c15ff3f5ba10f9f533cea93bdb0c36c556f16de9306"}
May 26 16:10:22.762204 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:10:22.762142    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-c7gdw" podStartSLOduration=1.3441539169999999 podStartE2EDuration="5.762124769s" podCreationTimestamp="2026-05-26 16:10:17 +0000 UTC" firstStartedPulling="2026-05-26 16:10:18.081115969 +0000 UTC m=+535.504709107" lastFinishedPulling="2026-05-26 16:10:22.499086806 +0000 UTC m=+539.922679959" observedRunningTime="2026-05-26 16:10:22.761449371 +0000 UTC m=+540.185042533" watchObservedRunningTime="2026-05-26 16:10:22.762124769 +0000 UTC m=+540.185717931"
May 26 16:11:23.054645 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:11:23.054615    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:11:23.055156 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:11:23.054831    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:14:23.648642 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:23.648608    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-c4j75"]
May 26 16:14:23.652013 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:23.651996    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-c4j75"
May 26 16:14:23.654248 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:23.654221    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 26 16:14:23.654408 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:23.654308    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 26 16:14:23.655194 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:23.655177    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 26 16:14:23.655284 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:23.655183    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-gv98l\""
May 26 16:14:23.657332 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:23.657306    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-c4j75"]
May 26 16:14:23.746722 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:23.746691    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5s2q\" (UniqueName: \"kubernetes.io/projected/d9ba0862-a9f0-4c34-866c-d45e2ceea18c-kube-api-access-f5s2q\") pod \"s3-init-c4j75\" (UID: \"d9ba0862-a9f0-4c34-866c-d45e2ceea18c\") " pod="kserve/s3-init-c4j75"
May 26 16:14:23.847878 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:23.847828    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f5s2q\" (UniqueName: \"kubernetes.io/projected/d9ba0862-a9f0-4c34-866c-d45e2ceea18c-kube-api-access-f5s2q\") pod \"s3-init-c4j75\" (UID: \"d9ba0862-a9f0-4c34-866c-d45e2ceea18c\") " pod="kserve/s3-init-c4j75"
May 26 16:14:23.856075 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:23.856047    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5s2q\" (UniqueName: \"kubernetes.io/projected/d9ba0862-a9f0-4c34-866c-d45e2ceea18c-kube-api-access-f5s2q\") pod \"s3-init-c4j75\" (UID: \"d9ba0862-a9f0-4c34-866c-d45e2ceea18c\") " pod="kserve/s3-init-c4j75"
May 26 16:14:23.971504 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:23.971423    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-c4j75"
May 26 16:14:24.096755 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:24.096726    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-c4j75"]
May 26 16:14:24.098691 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:14:24.098663    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9ba0862_a9f0_4c34_866c_d45e2ceea18c.slice/crio-561cadcd48b3b7cdf12a276d5f5cbef9e4957326a20436aacddf7ab21de27189 WatchSource:0}: Error finding container 561cadcd48b3b7cdf12a276d5f5cbef9e4957326a20436aacddf7ab21de27189: Status 404 returned error can't find the container with id 561cadcd48b3b7cdf12a276d5f5cbef9e4957326a20436aacddf7ab21de27189
May 26 16:14:24.437971 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:24.437939    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-c4j75" event={"ID":"d9ba0862-a9f0-4c34-866c-d45e2ceea18c","Type":"ContainerStarted","Data":"561cadcd48b3b7cdf12a276d5f5cbef9e4957326a20436aacddf7ab21de27189"}
May 26 16:14:29.455669 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:29.455630    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-c4j75" event={"ID":"d9ba0862-a9f0-4c34-866c-d45e2ceea18c","Type":"ContainerStarted","Data":"c61e510f2950feca0f88ecc99cc283d06ba01719ba684c014eeaf4777181d2b7"}
May 26 16:14:29.471512 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:29.471460    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-c4j75" podStartSLOduration=2.005727901 podStartE2EDuration="6.471444476s" podCreationTimestamp="2026-05-26 16:14:23 +0000 UTC" firstStartedPulling="2026-05-26 16:14:24.100403398 +0000 UTC m=+781.523996538" lastFinishedPulling="2026-05-26 16:14:28.566119968 +0000 UTC m=+785.989713113" observedRunningTime="2026-05-26 16:14:29.470542502 +0000 UTC m=+786.894135657" watchObservedRunningTime="2026-05-26 16:14:29.471444476 +0000 UTC m=+786.895037638"
May 26 16:14:32.468535 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:32.468505    2560 generic.go:358] "Generic (PLEG): container finished" podID="d9ba0862-a9f0-4c34-866c-d45e2ceea18c" containerID="c61e510f2950feca0f88ecc99cc283d06ba01719ba684c014eeaf4777181d2b7" exitCode=0
May 26 16:14:32.468914 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:32.468580    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-c4j75" event={"ID":"d9ba0862-a9f0-4c34-866c-d45e2ceea18c","Type":"ContainerDied","Data":"c61e510f2950feca0f88ecc99cc283d06ba01719ba684c014eeaf4777181d2b7"}
May 26 16:14:33.599721 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:33.599695    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-c4j75"
May 26 16:14:33.739453 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:33.739419    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5s2q\" (UniqueName: \"kubernetes.io/projected/d9ba0862-a9f0-4c34-866c-d45e2ceea18c-kube-api-access-f5s2q\") pod \"d9ba0862-a9f0-4c34-866c-d45e2ceea18c\" (UID: \"d9ba0862-a9f0-4c34-866c-d45e2ceea18c\") "
May 26 16:14:33.741540 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:33.741517    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ba0862-a9f0-4c34-866c-d45e2ceea18c-kube-api-access-f5s2q" (OuterVolumeSpecName: "kube-api-access-f5s2q") pod "d9ba0862-a9f0-4c34-866c-d45e2ceea18c" (UID: "d9ba0862-a9f0-4c34-866c-d45e2ceea18c"). InnerVolumeSpecName "kube-api-access-f5s2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:14:33.840272 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:33.840220    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5s2q\" (UniqueName: \"kubernetes.io/projected/d9ba0862-a9f0-4c34-866c-d45e2ceea18c-kube-api-access-f5s2q\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:14:34.476144 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:34.476109    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-c4j75" event={"ID":"d9ba0862-a9f0-4c34-866c-d45e2ceea18c","Type":"ContainerDied","Data":"561cadcd48b3b7cdf12a276d5f5cbef9e4957326a20436aacddf7ab21de27189"}
May 26 16:14:34.476144 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:34.476144    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="561cadcd48b3b7cdf12a276d5f5cbef9e4957326a20436aacddf7ab21de27189"
May 26 16:14:34.476144 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:34.476145    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-c4j75"
May 26 16:14:45.971923 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:45.971885    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"]
May 26 16:14:45.972413 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:45.972238    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ba0862-a9f0-4c34-866c-d45e2ceea18c" containerName="s3-init"
May 26 16:14:45.972413 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:45.972252    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ba0862-a9f0-4c34-866c-d45e2ceea18c" containerName="s3-init"
May 26 16:14:45.972413 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:45.972312    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="d9ba0862-a9f0-4c34-866c-d45e2ceea18c" containerName="s3-init"
May 26 16:14:45.978151 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:45.978129    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:14:45.980620 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:45.980600    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 16:14:45.981587 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:45.981571    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dzvrr\""
May 26 16:14:45.981684 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:45.981598    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 16:14:45.981745 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:45.981686    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
May 26 16:14:45.981745 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:45.981707    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
May 26 16:14:45.985376 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:45.985336    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"]
May 26 16:14:46.046098 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:46.046062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9a3c334-6524-46b8-b538-59f5a8327a1a-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-gshlx\" (UID: \"f9a3c334-6524-46b8-b538-59f5a8327a1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:14:46.046098 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:46.046099    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbrhs\" (UniqueName: \"kubernetes.io/projected/f9a3c334-6524-46b8-b538-59f5a8327a1a-kube-api-access-hbrhs\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-gshlx\" (UID: \"f9a3c334-6524-46b8-b538-59f5a8327a1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:14:46.046313 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:46.046198    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9a3c334-6524-46b8-b538-59f5a8327a1a-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-gshlx\" (UID: \"f9a3c334-6524-46b8-b538-59f5a8327a1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:14:46.046313 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:46.046248    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9a3c334-6524-46b8-b538-59f5a8327a1a-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-gshlx\" (UID: \"f9a3c334-6524-46b8-b538-59f5a8327a1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:14:46.147045 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:46.147014    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9a3c334-6524-46b8-b538-59f5a8327a1a-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-gshlx\" (UID: \"f9a3c334-6524-46b8-b538-59f5a8327a1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:14:46.147232 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:46.147056    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9a3c334-6524-46b8-b538-59f5a8327a1a-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-gshlx\" (UID: \"f9a3c334-6524-46b8-b538-59f5a8327a1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:14:46.147232 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:46.147075    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hbrhs\" (UniqueName: \"kubernetes.io/projected/f9a3c334-6524-46b8-b538-59f5a8327a1a-kube-api-access-hbrhs\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-gshlx\" (UID: \"f9a3c334-6524-46b8-b538-59f5a8327a1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:14:46.147232 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:46.147130    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9a3c334-6524-46b8-b538-59f5a8327a1a-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-gshlx\" (UID: \"f9a3c334-6524-46b8-b538-59f5a8327a1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:14:46.147506 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:46.147483    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9a3c334-6524-46b8-b538-59f5a8327a1a-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-gshlx\" (UID: \"f9a3c334-6524-46b8-b538-59f5a8327a1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:14:46.147754 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:46.147731    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9a3c334-6524-46b8-b538-59f5a8327a1a-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-gshlx\" (UID: \"f9a3c334-6524-46b8-b538-59f5a8327a1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:14:46.149523 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:46.149502    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9a3c334-6524-46b8-b538-59f5a8327a1a-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-gshlx\" (UID: \"f9a3c334-6524-46b8-b538-59f5a8327a1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:14:46.155612 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:46.155593    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbrhs\" (UniqueName: \"kubernetes.io/projected/f9a3c334-6524-46b8-b538-59f5a8327a1a-kube-api-access-hbrhs\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-gshlx\" (UID: \"f9a3c334-6524-46b8-b538-59f5a8327a1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:14:46.288614 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:46.288576    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:14:46.432849 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:46.432507    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"]
May 26 16:14:46.436973 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:14:46.436943    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a3c334_6524_46b8_b538_59f5a8327a1a.slice/crio-efc407b711936d53d28b8d2629f2c7de32042b8da015ad5bc951788ea646d2c3 WatchSource:0}: Error finding container efc407b711936d53d28b8d2629f2c7de32042b8da015ad5bc951788ea646d2c3: Status 404 returned error can't find the container with id efc407b711936d53d28b8d2629f2c7de32042b8da015ad5bc951788ea646d2c3
May 26 16:14:46.512913 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:46.512878    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" event={"ID":"f9a3c334-6524-46b8-b538-59f5a8327a1a","Type":"ContainerStarted","Data":"efc407b711936d53d28b8d2629f2c7de32042b8da015ad5bc951788ea646d2c3"}
May 26 16:14:47.033215 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.033180    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"]
May 26 16:14:47.037283 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.037256    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:14:47.039547 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.039523    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
May 26 16:14:47.039547 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.039539    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
May 26 16:14:47.051029 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.051008    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"]
May 26 16:14:47.053068 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.053043    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d528a173-7324-4e32-9844-a453d6c16eb2-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:14:47.053181 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.053165    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv2bl\" (UniqueName: \"kubernetes.io/projected/d528a173-7324-4e32-9844-a453d6c16eb2-kube-api-access-hv2bl\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:14:47.053245 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.053202    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d528a173-7324-4e32-9844-a453d6c16eb2-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:14:47.053302 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.053247    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d528a173-7324-4e32-9844-a453d6c16eb2-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:14:47.153831 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.153788    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d528a173-7324-4e32-9844-a453d6c16eb2-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:14:47.154008 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.153898    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hv2bl\" (UniqueName: \"kubernetes.io/projected/d528a173-7324-4e32-9844-a453d6c16eb2-kube-api-access-hv2bl\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:14:47.154008 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.153926    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d528a173-7324-4e32-9844-a453d6c16eb2-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:14:47.154008 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.153959    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d528a173-7324-4e32-9844-a453d6c16eb2-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:14:47.154008 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:14:47.153965    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
May 26 16:14:47.154214 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:14:47.154041    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d528a173-7324-4e32-9844-a453d6c16eb2-proxy-tls podName:d528a173-7324-4e32-9844-a453d6c16eb2 nodeName:}" failed. No retries permitted until 2026-05-26 16:14:47.654017729 +0000 UTC m=+805.077610869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d528a173-7324-4e32-9844-a453d6c16eb2-proxy-tls") pod "isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" (UID: "d528a173-7324-4e32-9844-a453d6c16eb2") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
May 26 16:14:47.154447 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.154425    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d528a173-7324-4e32-9844-a453d6c16eb2-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:14:47.154770 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.154748    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d528a173-7324-4e32-9844-a453d6c16eb2-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:14:47.162742 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.162718    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv2bl\" (UniqueName: \"kubernetes.io/projected/d528a173-7324-4e32-9844-a453d6c16eb2-kube-api-access-hv2bl\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:14:47.657370 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.657334    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d528a173-7324-4e32-9844-a453d6c16eb2-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:14:47.660116 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.660088    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d528a173-7324-4e32-9844-a453d6c16eb2-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:14:47.950777 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:47.950691    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:14:48.578219 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:48.578186    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"]
May 26 16:14:48.581731 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:14:48.581700    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd528a173_7324_4e32_9844_a453d6c16eb2.slice/crio-adb79e7f46d677b9db871bb3bdbbaa5d6a6f8bd9b7098125de154415a941f8ea WatchSource:0}: Error finding container adb79e7f46d677b9db871bb3bdbbaa5d6a6f8bd9b7098125de154415a941f8ea: Status 404 returned error can't find the container with id adb79e7f46d677b9db871bb3bdbbaa5d6a6f8bd9b7098125de154415a941f8ea
May 26 16:14:49.528362 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:49.528300    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" event={"ID":"d528a173-7324-4e32-9844-a453d6c16eb2","Type":"ContainerStarted","Data":"adb79e7f46d677b9db871bb3bdbbaa5d6a6f8bd9b7098125de154415a941f8ea"}
May 26 16:14:50.540939 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:50.540861    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" event={"ID":"f9a3c334-6524-46b8-b538-59f5a8327a1a","Type":"ContainerStarted","Data":"5156f1a04a25f14fa3ac70bcf4b7d309c9b8e555953621f6744508c646530430"}
May 26 16:14:50.542435 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:50.542406    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" event={"ID":"d528a173-7324-4e32-9844-a453d6c16eb2","Type":"ContainerStarted","Data":"5ce334709f6a0b314e44163876c441223b710811d5a7062a23c5fe32702a6d54"}
May 26 16:14:54.555343 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:54.555307    2560 generic.go:358] "Generic (PLEG): container finished" podID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerID="5156f1a04a25f14fa3ac70bcf4b7d309c9b8e555953621f6744508c646530430" exitCode=0
May 26 16:14:54.555875 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:54.555384    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" event={"ID":"f9a3c334-6524-46b8-b538-59f5a8327a1a","Type":"ContainerDied","Data":"5156f1a04a25f14fa3ac70bcf4b7d309c9b8e555953621f6744508c646530430"}
May 26 16:14:54.556693 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:54.556674    2560 generic.go:358] "Generic (PLEG): container finished" podID="d528a173-7324-4e32-9844-a453d6c16eb2" containerID="5ce334709f6a0b314e44163876c441223b710811d5a7062a23c5fe32702a6d54" exitCode=0
May 26 16:14:54.556766 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:14:54.556746    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" event={"ID":"d528a173-7324-4e32-9844-a453d6c16eb2","Type":"ContainerDied","Data":"5ce334709f6a0b314e44163876c441223b710811d5a7062a23c5fe32702a6d54"}
May 26 16:15:09.618318 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:09.618274    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" event={"ID":"d528a173-7324-4e32-9844-a453d6c16eb2","Type":"ContainerStarted","Data":"52fcc5571582f1d0fda634bbd3e0fb985a3e245613323dcd3805285f73612718"}
May 26 16:15:09.620480 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:09.620448    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" event={"ID":"f9a3c334-6524-46b8-b538-59f5a8327a1a","Type":"ContainerStarted","Data":"d5df28d1d23a189919f9f2c76912a93ad33b00038a71d96de9eb5a1f6b57893c"}
May 26 16:15:11.628784 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:11.628747    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" event={"ID":"d528a173-7324-4e32-9844-a453d6c16eb2","Type":"ContainerStarted","Data":"ff6031b8c6594d42e4bdd880744a1e1559b807312995b7aff543371d0875fa71"}
May 26 16:15:11.629239 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:11.628848    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:15:11.630606 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:11.630582    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" event={"ID":"f9a3c334-6524-46b8-b538-59f5a8327a1a","Type":"ContainerStarted","Data":"7c8fb3a11de680ba2b2242b0a13424978f4a8072f65cbedca5f20803b4296ca5"}
May 26 16:15:11.630870 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:11.630853    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:15:11.630910 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:11.630878    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:15:11.632050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:11.632027    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 26 16:15:11.647362 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:11.647317    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" podStartSLOduration=1.7034146030000001 podStartE2EDuration="24.647306069s" podCreationTimestamp="2026-05-26 16:14:47 +0000 UTC" firstStartedPulling="2026-05-26 16:14:48.584213761 +0000 UTC m=+806.007806907" lastFinishedPulling="2026-05-26 16:15:11.528105232 +0000 UTC m=+828.951698373" observedRunningTime="2026-05-26 16:15:11.645109031 +0000 UTC m=+829.068702192" watchObservedRunningTime="2026-05-26 16:15:11.647306069 +0000 UTC m=+829.070899229"
May 26 16:15:11.662359 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:11.662319    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" podStartSLOduration=1.576232297 podStartE2EDuration="26.662306374s" podCreationTimestamp="2026-05-26 16:14:45 +0000 UTC" firstStartedPulling="2026-05-26 16:14:46.439007515 +0000 UTC m=+803.862600655" lastFinishedPulling="2026-05-26 16:15:11.525081578 +0000 UTC m=+828.948674732" observedRunningTime="2026-05-26 16:15:11.661775478 +0000 UTC m=+829.085368639" watchObservedRunningTime="2026-05-26 16:15:11.662306374 +0000 UTC m=+829.085899568"
May 26 16:15:12.634683 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:12.634634    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 26 16:15:12.635136 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:12.634837    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:15:12.635891 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:12.635866    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 26 16:15:13.637141 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:13.637106    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 26 16:15:17.639469 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:17.639437    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:15:17.640052 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:17.640027    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 26 16:15:18.641280 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:18.641250    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:15:18.641799 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:18.641768    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 26 16:15:27.640375 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:27.640337    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 26 16:15:28.641766 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:28.641727    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 26 16:15:37.640919 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:37.640879    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 26 16:15:38.642578 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:38.642541    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 26 16:15:47.640516 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:47.640475    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 26 16:15:48.642368 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:48.642331    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 26 16:15:57.640034 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:57.639996    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 26 16:15:58.642660 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:15:58.642623    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 26 16:16:07.640687 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:07.640601    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 26 16:16:08.641903 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:08.641863    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 26 16:16:17.640592 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:17.640561    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:16:18.643119 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:18.643085    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:16:23.076983 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:23.076955    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:16:23.077783 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:23.077765    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:16:56.007458 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:56.007419    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"]
May 26 16:16:56.008063 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:56.007825    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kserve-container" containerID="cri-o://52fcc5571582f1d0fda634bbd3e0fb985a3e245613323dcd3805285f73612718" gracePeriod=30
May 26 16:16:56.008141 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:56.008077    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kube-rbac-proxy" containerID="cri-o://ff6031b8c6594d42e4bdd880744a1e1559b807312995b7aff543371d0875fa71" gracePeriod=30
May 26 16:16:56.063698 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:56.063671    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"]
May 26 16:16:56.064020 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:56.063990    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kserve-container" containerID="cri-o://d5df28d1d23a189919f9f2c76912a93ad33b00038a71d96de9eb5a1f6b57893c" gracePeriod=30
May 26 16:16:56.064020 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:56.063998    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kube-rbac-proxy" containerID="cri-o://7c8fb3a11de680ba2b2242b0a13424978f4a8072f65cbedca5f20803b4296ca5" gracePeriod=30
May 26 16:16:56.960083 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:56.960051    2560 generic.go:358] "Generic (PLEG): container finished" podID="d528a173-7324-4e32-9844-a453d6c16eb2" containerID="ff6031b8c6594d42e4bdd880744a1e1559b807312995b7aff543371d0875fa71" exitCode=2
May 26 16:16:56.960273 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:56.960121    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" event={"ID":"d528a173-7324-4e32-9844-a453d6c16eb2","Type":"ContainerDied","Data":"ff6031b8c6594d42e4bdd880744a1e1559b807312995b7aff543371d0875fa71"}
May 26 16:16:56.961714 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:56.961692    2560 generic.go:358] "Generic (PLEG): container finished" podID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerID="7c8fb3a11de680ba2b2242b0a13424978f4a8072f65cbedca5f20803b4296ca5" exitCode=2
May 26 16:16:56.961825 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:56.961730    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" event={"ID":"f9a3c334-6524-46b8-b538-59f5a8327a1a","Type":"ContainerDied","Data":"7c8fb3a11de680ba2b2242b0a13424978f4a8072f65cbedca5f20803b4296ca5"}
May 26 16:16:57.635706 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:57.635664    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
May 26 16:16:57.640084 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:57.640059    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 26 16:16:58.637621 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:58.637580    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
May 26 16:16:58.641974 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:16:58.641953    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 26 16:17:00.415139 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.415113    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:17:00.452901 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.452877    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:17:00.470332 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.470306    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbrhs\" (UniqueName: \"kubernetes.io/projected/f9a3c334-6524-46b8-b538-59f5a8327a1a-kube-api-access-hbrhs\") pod \"f9a3c334-6524-46b8-b538-59f5a8327a1a\" (UID: \"f9a3c334-6524-46b8-b538-59f5a8327a1a\") "
May 26 16:17:00.470486 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.470450    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9a3c334-6524-46b8-b538-59f5a8327a1a-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"f9a3c334-6524-46b8-b538-59f5a8327a1a\" (UID: \"f9a3c334-6524-46b8-b538-59f5a8327a1a\") "
May 26 16:17:00.470600 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.470578    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9a3c334-6524-46b8-b538-59f5a8327a1a-kserve-provision-location\") pod \"f9a3c334-6524-46b8-b538-59f5a8327a1a\" (UID: \"f9a3c334-6524-46b8-b538-59f5a8327a1a\") "
May 26 16:17:00.471054 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.470755    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9a3c334-6524-46b8-b538-59f5a8327a1a-proxy-tls\") pod \"f9a3c334-6524-46b8-b538-59f5a8327a1a\" (UID: \"f9a3c334-6524-46b8-b538-59f5a8327a1a\") "
May 26 16:17:00.471054 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.470964    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a3c334-6524-46b8-b538-59f5a8327a1a-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-1-kube-rbac-proxy-sar-config") pod "f9a3c334-6524-46b8-b538-59f5a8327a1a" (UID: "f9a3c334-6524-46b8-b538-59f5a8327a1a"). InnerVolumeSpecName "isvc-sklearn-graph-1-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:17:00.471054 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.471033    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a3c334-6524-46b8-b538-59f5a8327a1a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9a3c334-6524-46b8-b538-59f5a8327a1a" (UID: "f9a3c334-6524-46b8-b538-59f5a8327a1a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:17:00.471054 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.471038    2560 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9a3c334-6524-46b8-b538-59f5a8327a1a-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:17:00.473218 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.473195    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9a3c334-6524-46b8-b538-59f5a8327a1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9a3c334-6524-46b8-b538-59f5a8327a1a" (UID: "f9a3c334-6524-46b8-b538-59f5a8327a1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:17:00.473296 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.473235    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a3c334-6524-46b8-b538-59f5a8327a1a-kube-api-access-hbrhs" (OuterVolumeSpecName: "kube-api-access-hbrhs") pod "f9a3c334-6524-46b8-b538-59f5a8327a1a" (UID: "f9a3c334-6524-46b8-b538-59f5a8327a1a"). InnerVolumeSpecName "kube-api-access-hbrhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:17:00.571801 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.571760    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d528a173-7324-4e32-9844-a453d6c16eb2-kserve-provision-location\") pod \"d528a173-7324-4e32-9844-a453d6c16eb2\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") "
May 26 16:17:00.571801 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.571808    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d528a173-7324-4e32-9844-a453d6c16eb2-proxy-tls\") pod \"d528a173-7324-4e32-9844-a453d6c16eb2\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") "
May 26 16:17:00.572005 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.571837    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv2bl\" (UniqueName: \"kubernetes.io/projected/d528a173-7324-4e32-9844-a453d6c16eb2-kube-api-access-hv2bl\") pod \"d528a173-7324-4e32-9844-a453d6c16eb2\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") "
May 26 16:17:00.572005 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.571899    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d528a173-7324-4e32-9844-a453d6c16eb2-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"d528a173-7324-4e32-9844-a453d6c16eb2\" (UID: \"d528a173-7324-4e32-9844-a453d6c16eb2\") "
May 26 16:17:00.572107 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.572039    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9a3c334-6524-46b8-b538-59f5a8327a1a-proxy-tls\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:17:00.572107 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.572050    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbrhs\" (UniqueName: \"kubernetes.io/projected/f9a3c334-6524-46b8-b538-59f5a8327a1a-kube-api-access-hbrhs\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:17:00.572107 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.572060    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9a3c334-6524-46b8-b538-59f5a8327a1a-kserve-provision-location\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:17:00.572266 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.572122    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d528a173-7324-4e32-9844-a453d6c16eb2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d528a173-7324-4e32-9844-a453d6c16eb2" (UID: "d528a173-7324-4e32-9844-a453d6c16eb2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:17:00.572356 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.572330    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d528a173-7324-4e32-9844-a453d6c16eb2-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-2-kube-rbac-proxy-sar-config") pod "d528a173-7324-4e32-9844-a453d6c16eb2" (UID: "d528a173-7324-4e32-9844-a453d6c16eb2"). InnerVolumeSpecName "isvc-sklearn-graph-2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:17:00.573837 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.573818    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d528a173-7324-4e32-9844-a453d6c16eb2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d528a173-7324-4e32-9844-a453d6c16eb2" (UID: "d528a173-7324-4e32-9844-a453d6c16eb2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:17:00.573892 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.573855    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d528a173-7324-4e32-9844-a453d6c16eb2-kube-api-access-hv2bl" (OuterVolumeSpecName: "kube-api-access-hv2bl") pod "d528a173-7324-4e32-9844-a453d6c16eb2" (UID: "d528a173-7324-4e32-9844-a453d6c16eb2"). InnerVolumeSpecName "kube-api-access-hv2bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:17:00.672727 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.672696    2560 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d528a173-7324-4e32-9844-a453d6c16eb2-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:17:00.672727 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.672723    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d528a173-7324-4e32-9844-a453d6c16eb2-kserve-provision-location\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:17:00.672727 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.672735    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d528a173-7324-4e32-9844-a453d6c16eb2-proxy-tls\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:17:00.672931 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.672744    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv2bl\" (UniqueName: \"kubernetes.io/projected/d528a173-7324-4e32-9844-a453d6c16eb2-kube-api-access-hv2bl\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:17:00.975509 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.975474    2560 generic.go:358] "Generic (PLEG): container finished" podID="d528a173-7324-4e32-9844-a453d6c16eb2" containerID="52fcc5571582f1d0fda634bbd3e0fb985a3e245613323dcd3805285f73612718" exitCode=0
May 26 16:17:00.975689 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.975560    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" event={"ID":"d528a173-7324-4e32-9844-a453d6c16eb2","Type":"ContainerDied","Data":"52fcc5571582f1d0fda634bbd3e0fb985a3e245613323dcd3805285f73612718"}
May 26 16:17:00.975689 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.975569    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"
May 26 16:17:00.975689 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.975597    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc" event={"ID":"d528a173-7324-4e32-9844-a453d6c16eb2","Type":"ContainerDied","Data":"adb79e7f46d677b9db871bb3bdbbaa5d6a6f8bd9b7098125de154415a941f8ea"}
May 26 16:17:00.975689 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.975612    2560 scope.go:117] "RemoveContainer" containerID="ff6031b8c6594d42e4bdd880744a1e1559b807312995b7aff543371d0875fa71"
May 26 16:17:00.977225 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.977205    2560 generic.go:358] "Generic (PLEG): container finished" podID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerID="d5df28d1d23a189919f9f2c76912a93ad33b00038a71d96de9eb5a1f6b57893c" exitCode=0
May 26 16:17:00.977362 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.977242    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" event={"ID":"f9a3c334-6524-46b8-b538-59f5a8327a1a","Type":"ContainerDied","Data":"d5df28d1d23a189919f9f2c76912a93ad33b00038a71d96de9eb5a1f6b57893c"}
May 26 16:17:00.977362 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.977262    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx" event={"ID":"f9a3c334-6524-46b8-b538-59f5a8327a1a","Type":"ContainerDied","Data":"efc407b711936d53d28b8d2629f2c7de32042b8da015ad5bc951788ea646d2c3"}
May 26 16:17:00.977362 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.977294    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"
May 26 16:17:00.991904 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.991873    2560 scope.go:117] "RemoveContainer" containerID="52fcc5571582f1d0fda634bbd3e0fb985a3e245613323dcd3805285f73612718"
May 26 16:17:00.999198 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:00.999183    2560 scope.go:117] "RemoveContainer" containerID="5ce334709f6a0b314e44163876c441223b710811d5a7062a23c5fe32702a6d54"
May 26 16:17:01.007158 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.007141    2560 scope.go:117] "RemoveContainer" containerID="ff6031b8c6594d42e4bdd880744a1e1559b807312995b7aff543371d0875fa71"
May 26 16:17:01.007328 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.007312    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"]
May 26 16:17:01.007428 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:17:01.007413    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff6031b8c6594d42e4bdd880744a1e1559b807312995b7aff543371d0875fa71\": container with ID starting with ff6031b8c6594d42e4bdd880744a1e1559b807312995b7aff543371d0875fa71 not found: ID does not exist" containerID="ff6031b8c6594d42e4bdd880744a1e1559b807312995b7aff543371d0875fa71"
May 26 16:17:01.007467 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.007434    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff6031b8c6594d42e4bdd880744a1e1559b807312995b7aff543371d0875fa71"} err="failed to get container status \"ff6031b8c6594d42e4bdd880744a1e1559b807312995b7aff543371d0875fa71\": rpc error: code = NotFound desc = could not find container \"ff6031b8c6594d42e4bdd880744a1e1559b807312995b7aff543371d0875fa71\": container with ID starting with ff6031b8c6594d42e4bdd880744a1e1559b807312995b7aff543371d0875fa71 not found: ID does not exist"
May 26 16:17:01.007467 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.007451    2560 scope.go:117] "RemoveContainer" containerID="52fcc5571582f1d0fda634bbd3e0fb985a3e245613323dcd3805285f73612718"
May 26 16:17:01.007665 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:17:01.007651    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52fcc5571582f1d0fda634bbd3e0fb985a3e245613323dcd3805285f73612718\": container with ID starting with 52fcc5571582f1d0fda634bbd3e0fb985a3e245613323dcd3805285f73612718 not found: ID does not exist" containerID="52fcc5571582f1d0fda634bbd3e0fb985a3e245613323dcd3805285f73612718"
May 26 16:17:01.007700 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.007669    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52fcc5571582f1d0fda634bbd3e0fb985a3e245613323dcd3805285f73612718"} err="failed to get container status \"52fcc5571582f1d0fda634bbd3e0fb985a3e245613323dcd3805285f73612718\": rpc error: code = NotFound desc = could not find container \"52fcc5571582f1d0fda634bbd3e0fb985a3e245613323dcd3805285f73612718\": container with ID starting with 52fcc5571582f1d0fda634bbd3e0fb985a3e245613323dcd3805285f73612718 not found: ID does not exist"
May 26 16:17:01.007700 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.007681    2560 scope.go:117] "RemoveContainer" containerID="5ce334709f6a0b314e44163876c441223b710811d5a7062a23c5fe32702a6d54"
May 26 16:17:01.007846 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:17:01.007834    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ce334709f6a0b314e44163876c441223b710811d5a7062a23c5fe32702a6d54\": container with ID starting with 5ce334709f6a0b314e44163876c441223b710811d5a7062a23c5fe32702a6d54 not found: ID does not exist" containerID="5ce334709f6a0b314e44163876c441223b710811d5a7062a23c5fe32702a6d54"
May 26 16:17:01.007881 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.007849    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce334709f6a0b314e44163876c441223b710811d5a7062a23c5fe32702a6d54"} err="failed to get container status \"5ce334709f6a0b314e44163876c441223b710811d5a7062a23c5fe32702a6d54\": rpc error: code = NotFound desc = could not find container \"5ce334709f6a0b314e44163876c441223b710811d5a7062a23c5fe32702a6d54\": container with ID starting with 5ce334709f6a0b314e44163876c441223b710811d5a7062a23c5fe32702a6d54 not found: ID does not exist"
May 26 16:17:01.007881 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.007859    2560 scope.go:117] "RemoveContainer" containerID="7c8fb3a11de680ba2b2242b0a13424978f4a8072f65cbedca5f20803b4296ca5"
May 26 16:17:01.012271 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.012251    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-n7tbc"]
May 26 16:17:01.014854 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.014836    2560 scope.go:117] "RemoveContainer" containerID="d5df28d1d23a189919f9f2c76912a93ad33b00038a71d96de9eb5a1f6b57893c"
May 26 16:17:01.021225 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.021208    2560 scope.go:117] "RemoveContainer" containerID="5156f1a04a25f14fa3ac70bcf4b7d309c9b8e555953621f6744508c646530430"
May 26 16:17:01.024564 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.024536    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"]
May 26 16:17:01.028520 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.028499    2560 scope.go:117] "RemoveContainer" containerID="7c8fb3a11de680ba2b2242b0a13424978f4a8072f65cbedca5f20803b4296ca5"
May 26 16:17:01.028769 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:17:01.028750    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8fb3a11de680ba2b2242b0a13424978f4a8072f65cbedca5f20803b4296ca5\": container with ID starting with 7c8fb3a11de680ba2b2242b0a13424978f4a8072f65cbedca5f20803b4296ca5 not found: ID does not exist" containerID="7c8fb3a11de680ba2b2242b0a13424978f4a8072f65cbedca5f20803b4296ca5"
May 26 16:17:01.028854 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.028784    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8fb3a11de680ba2b2242b0a13424978f4a8072f65cbedca5f20803b4296ca5"} err="failed to get container status \"7c8fb3a11de680ba2b2242b0a13424978f4a8072f65cbedca5f20803b4296ca5\": rpc error: code = NotFound desc = could not find container \"7c8fb3a11de680ba2b2242b0a13424978f4a8072f65cbedca5f20803b4296ca5\": container with ID starting with 7c8fb3a11de680ba2b2242b0a13424978f4a8072f65cbedca5f20803b4296ca5 not found: ID does not exist"
May 26 16:17:01.028854 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.028801    2560 scope.go:117] "RemoveContainer" containerID="d5df28d1d23a189919f9f2c76912a93ad33b00038a71d96de9eb5a1f6b57893c"
May 26 16:17:01.029020 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:17:01.029004    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5df28d1d23a189919f9f2c76912a93ad33b00038a71d96de9eb5a1f6b57893c\": container with ID starting with d5df28d1d23a189919f9f2c76912a93ad33b00038a71d96de9eb5a1f6b57893c not found: ID does not exist" containerID="d5df28d1d23a189919f9f2c76912a93ad33b00038a71d96de9eb5a1f6b57893c"
May 26 16:17:01.029062 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.029029    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5df28d1d23a189919f9f2c76912a93ad33b00038a71d96de9eb5a1f6b57893c"} err="failed to get container status \"d5df28d1d23a189919f9f2c76912a93ad33b00038a71d96de9eb5a1f6b57893c\": rpc error: code = NotFound desc = could not find container \"d5df28d1d23a189919f9f2c76912a93ad33b00038a71d96de9eb5a1f6b57893c\": container with ID starting with d5df28d1d23a189919f9f2c76912a93ad33b00038a71d96de9eb5a1f6b57893c not found: ID does not exist"
May 26 16:17:01.029062 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.029043    2560 scope.go:117] "RemoveContainer" containerID="5156f1a04a25f14fa3ac70bcf4b7d309c9b8e555953621f6744508c646530430"
May 26 16:17:01.029247 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:17:01.029231    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5156f1a04a25f14fa3ac70bcf4b7d309c9b8e555953621f6744508c646530430\": container with ID starting with 5156f1a04a25f14fa3ac70bcf4b7d309c9b8e555953621f6744508c646530430 not found: ID does not exist" containerID="5156f1a04a25f14fa3ac70bcf4b7d309c9b8e555953621f6744508c646530430"
May 26 16:17:01.029299 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.029249    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5156f1a04a25f14fa3ac70bcf4b7d309c9b8e555953621f6744508c646530430"} err="failed to get container status \"5156f1a04a25f14fa3ac70bcf4b7d309c9b8e555953621f6744508c646530430\": rpc error: code = NotFound desc = could not find container \"5156f1a04a25f14fa3ac70bcf4b7d309c9b8e555953621f6744508c646530430\": container with ID starting with 5156f1a04a25f14fa3ac70bcf4b7d309c9b8e555953621f6744508c646530430 not found: ID does not exist"
May 26 16:17:01.029864 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.029846    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-gshlx"]
May 26 16:17:01.170724 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.170690    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" path="/var/lib/kubelet/pods/d528a173-7324-4e32-9844-a453d6c16eb2/volumes"
May 26 16:17:01.171220 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:01.171201    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" path="/var/lib/kubelet/pods/f9a3c334-6524-46b8-b538-59f5a8327a1a/volumes"
May 26 16:17:56.081909 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.081823    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"]
May 26 16:17:56.082468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.082214    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="storage-initializer"
May 26 16:17:56.082468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.082228    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="storage-initializer"
May 26 16:17:56.082468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.082242    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kube-rbac-proxy"
May 26 16:17:56.082468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.082250    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kube-rbac-proxy"
May 26 16:17:56.082468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.082262    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kube-rbac-proxy"
May 26 16:17:56.082468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.082268    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kube-rbac-proxy"
May 26 16:17:56.082468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.082280    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="storage-initializer"
May 26 16:17:56.082468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.082286    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="storage-initializer"
May 26 16:17:56.082468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.082291    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kserve-container"
May 26 16:17:56.082468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.082296    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kserve-container"
May 26 16:17:56.082468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.082307    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kserve-container"
May 26 16:17:56.082468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.082314    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kserve-container"
May 26 16:17:56.082468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.082399    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kserve-container"
May 26 16:17:56.082468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.082410    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="d528a173-7324-4e32-9844-a453d6c16eb2" containerName="kube-rbac-proxy"
May 26 16:17:56.082468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.082418    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kube-rbac-proxy"
May 26 16:17:56.082468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.082424    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="f9a3c334-6524-46b8-b538-59f5a8327a1a" containerName="kserve-container"
May 26 16:17:56.085163 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.085148    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"
May 26 16:17:56.087510 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.087485    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-ad62d-kube-rbac-proxy-sar-config\""
May 26 16:17:56.087638 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.087497    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-ad62d-serving-cert\""
May 26 16:17:56.087638 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.087579    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dzvrr\""
May 26 16:17:56.087638 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.087594    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 16:17:56.093006 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.092987    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"]
May 26 16:17:56.131251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.131224    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11101453-8b37-40af-bdbc-76db842bb667-openshift-service-ca-bundle\") pod \"sequence-graph-ad62d-6db94b859c-qz87l\" (UID: \"11101453-8b37-40af-bdbc-76db842bb667\") " pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"
May 26 16:17:56.131447 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.131330    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11101453-8b37-40af-bdbc-76db842bb667-proxy-tls\") pod \"sequence-graph-ad62d-6db94b859c-qz87l\" (UID: \"11101453-8b37-40af-bdbc-76db842bb667\") " pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"
May 26 16:17:56.232355 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.232326    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11101453-8b37-40af-bdbc-76db842bb667-proxy-tls\") pod \"sequence-graph-ad62d-6db94b859c-qz87l\" (UID: \"11101453-8b37-40af-bdbc-76db842bb667\") " pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"
May 26 16:17:56.232527 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.232431    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11101453-8b37-40af-bdbc-76db842bb667-openshift-service-ca-bundle\") pod \"sequence-graph-ad62d-6db94b859c-qz87l\" (UID: \"11101453-8b37-40af-bdbc-76db842bb667\") " pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"
May 26 16:17:56.232527 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:17:56.232477    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-ad62d-serving-cert: secret "sequence-graph-ad62d-serving-cert" not found
May 26 16:17:56.232618 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:17:56.232540    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11101453-8b37-40af-bdbc-76db842bb667-proxy-tls podName:11101453-8b37-40af-bdbc-76db842bb667 nodeName:}" failed. No retries permitted until 2026-05-26 16:17:56.732522152 +0000 UTC m=+994.156115291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/11101453-8b37-40af-bdbc-76db842bb667-proxy-tls") pod "sequence-graph-ad62d-6db94b859c-qz87l" (UID: "11101453-8b37-40af-bdbc-76db842bb667") : secret "sequence-graph-ad62d-serving-cert" not found
May 26 16:17:56.233121 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.233102    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11101453-8b37-40af-bdbc-76db842bb667-openshift-service-ca-bundle\") pod \"sequence-graph-ad62d-6db94b859c-qz87l\" (UID: \"11101453-8b37-40af-bdbc-76db842bb667\") " pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"
May 26 16:17:56.736675 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.736629    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11101453-8b37-40af-bdbc-76db842bb667-proxy-tls\") pod \"sequence-graph-ad62d-6db94b859c-qz87l\" (UID: \"11101453-8b37-40af-bdbc-76db842bb667\") " pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"
May 26 16:17:56.739010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.738984    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11101453-8b37-40af-bdbc-76db842bb667-proxy-tls\") pod \"sequence-graph-ad62d-6db94b859c-qz87l\" (UID: \"11101453-8b37-40af-bdbc-76db842bb667\") " pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"
May 26 16:17:56.995464 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:56.995354    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"
May 26 16:17:57.116184 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:57.116154    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"]
May 26 16:17:57.118925 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:17:57.118891    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11101453_8b37_40af_bdbc_76db842bb667.slice/crio-f70fd7bff4be2af4504301397b2f049f4ef5f2bcd2a87b0fae7725b0aecfd0ec WatchSource:0}: Error finding container f70fd7bff4be2af4504301397b2f049f4ef5f2bcd2a87b0fae7725b0aecfd0ec: Status 404 returned error can't find the container with id f70fd7bff4be2af4504301397b2f049f4ef5f2bcd2a87b0fae7725b0aecfd0ec
May 26 16:17:57.121052 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:57.121035    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:17:57.139251 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:57.139225    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l" event={"ID":"11101453-8b37-40af-bdbc-76db842bb667","Type":"ContainerStarted","Data":"f70fd7bff4be2af4504301397b2f049f4ef5f2bcd2a87b0fae7725b0aecfd0ec"}
May 26 16:17:59.145968 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:59.145933    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l" event={"ID":"11101453-8b37-40af-bdbc-76db842bb667","Type":"ContainerStarted","Data":"415c17be8d1594c4c6dff4babbea68112452f480949ec2d6bcf2d1ee5c4be422"}
May 26 16:17:59.146450 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:59.146054    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"
May 26 16:17:59.163331 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:17:59.163279    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l" podStartSLOduration=1.304660155 podStartE2EDuration="3.163266385s" podCreationTimestamp="2026-05-26 16:17:56 +0000 UTC" firstStartedPulling="2026-05-26 16:17:57.121184068 +0000 UTC m=+994.544777207" lastFinishedPulling="2026-05-26 16:17:58.979790298 +0000 UTC m=+996.403383437" observedRunningTime="2026-05-26 16:17:59.161876569 +0000 UTC m=+996.585469731" watchObservedRunningTime="2026-05-26 16:17:59.163266385 +0000 UTC m=+996.586859546"
May 26 16:18:05.155080 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:18:05.155053    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"
May 26 16:21:23.098265 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:21:23.098235    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:21:23.100108 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:21:23.100088    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:26:10.825516 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:10.825474    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"]
May 26 16:26:10.826086 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:10.825779    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l" podUID="11101453-8b37-40af-bdbc-76db842bb667" containerName="sequence-graph-ad62d" containerID="cri-o://415c17be8d1594c4c6dff4babbea68112452f480949ec2d6bcf2d1ee5c4be422" gracePeriod=30
May 26 16:26:15.152799 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:15.152752    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l" podUID="11101453-8b37-40af-bdbc-76db842bb667" containerName="sequence-graph-ad62d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:26:20.152928 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:20.152888    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l" podUID="11101453-8b37-40af-bdbc-76db842bb667" containerName="sequence-graph-ad62d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:26:23.119571 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:23.119535    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:26:23.121668 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:23.121643    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:26:25.153048 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:25.153009    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l" podUID="11101453-8b37-40af-bdbc-76db842bb667" containerName="sequence-graph-ad62d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:26:25.153476 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:25.153157    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"
May 26 16:26:30.153629 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:30.153585    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l" podUID="11101453-8b37-40af-bdbc-76db842bb667" containerName="sequence-graph-ad62d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:26:31.163648 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:31.163610    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"]
May 26 16:26:31.167080 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:31.167053    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"
May 26 16:26:31.169468 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:31.169441    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-007a5-kube-rbac-proxy-sar-config\""
May 26 16:26:31.169576 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:31.169446    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-007a5-serving-cert\""
May 26 16:26:31.175575 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:31.175554    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"]
May 26 16:26:31.254898 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:31.254861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0-openshift-service-ca-bundle\") pod \"ensemble-graph-007a5-866fd95dbf-w2kn6\" (UID: \"13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0\") " pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"
May 26 16:26:31.255063 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:31.254912    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0-proxy-tls\") pod \"ensemble-graph-007a5-866fd95dbf-w2kn6\" (UID: \"13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0\") " pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"
May 26 16:26:31.356362 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:31.356325    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0-proxy-tls\") pod \"ensemble-graph-007a5-866fd95dbf-w2kn6\" (UID: \"13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0\") " pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"
May 26 16:26:31.356554 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:31.356433    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0-openshift-service-ca-bundle\") pod \"ensemble-graph-007a5-866fd95dbf-w2kn6\" (UID: \"13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0\") " pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"
May 26 16:26:31.356554 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:26:31.356523    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-007a5-serving-cert: secret "ensemble-graph-007a5-serving-cert" not found
May 26 16:26:31.356633 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:26:31.356608    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0-proxy-tls podName:13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0 nodeName:}" failed. No retries permitted until 2026-05-26 16:26:31.856583918 +0000 UTC m=+1509.280177064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0-proxy-tls") pod "ensemble-graph-007a5-866fd95dbf-w2kn6" (UID: "13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0") : secret "ensemble-graph-007a5-serving-cert" not found
May 26 16:26:31.357017 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:31.356999    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0-openshift-service-ca-bundle\") pod \"ensemble-graph-007a5-866fd95dbf-w2kn6\" (UID: \"13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0\") " pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"
May 26 16:26:31.860320 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:31.860282    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0-proxy-tls\") pod \"ensemble-graph-007a5-866fd95dbf-w2kn6\" (UID: \"13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0\") " pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"
May 26 16:26:31.862777 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:31.862753    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0-proxy-tls\") pod \"ensemble-graph-007a5-866fd95dbf-w2kn6\" (UID: \"13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0\") " pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"
May 26 16:26:32.078238 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:32.078188    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"
May 26 16:26:32.204656 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:32.204630    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"]
May 26 16:26:32.207093 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:26:32.207058    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f28c0b_3a9c_4fc3_b1f9_67f11f2be0a0.slice/crio-8431f562df9b8f2b4aab96f807d0d252bd795fb0f15d1d1478e91141a513ce29 WatchSource:0}: Error finding container 8431f562df9b8f2b4aab96f807d0d252bd795fb0f15d1d1478e91141a513ce29: Status 404 returned error can't find the container with id 8431f562df9b8f2b4aab96f807d0d252bd795fb0f15d1d1478e91141a513ce29
May 26 16:26:32.208914 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:32.208898    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:26:32.635362 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:32.635327    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6" event={"ID":"13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0","Type":"ContainerStarted","Data":"c7d7d449bb60ff724142f00fc09e2ea8b59cac120c7d6d916e92b301326c225f"}
May 26 16:26:32.635362 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:32.635366    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6" event={"ID":"13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0","Type":"ContainerStarted","Data":"8431f562df9b8f2b4aab96f807d0d252bd795fb0f15d1d1478e91141a513ce29"}
May 26 16:26:32.635584 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:32.635505    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"
May 26 16:26:32.652229 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:32.652179    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6" podStartSLOduration=1.652164481 podStartE2EDuration="1.652164481s" podCreationTimestamp="2026-05-26 16:26:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:26:32.651749365 +0000 UTC m=+1510.075342526" watchObservedRunningTime="2026-05-26 16:26:32.652164481 +0000 UTC m=+1510.075757698"
May 26 16:26:35.152979 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:35.152894    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l" podUID="11101453-8b37-40af-bdbc-76db842bb667" containerName="sequence-graph-ad62d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:26:38.644221 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:38.644190    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"
May 26 16:26:40.153456 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:40.153411    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l" podUID="11101453-8b37-40af-bdbc-76db842bb667" containerName="sequence-graph-ad62d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:26:40.976678 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:40.976656    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"
May 26 16:26:41.039278 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.039240    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11101453-8b37-40af-bdbc-76db842bb667-proxy-tls\") pod \"11101453-8b37-40af-bdbc-76db842bb667\" (UID: \"11101453-8b37-40af-bdbc-76db842bb667\") "
May 26 16:26:41.039461 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.039299    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11101453-8b37-40af-bdbc-76db842bb667-openshift-service-ca-bundle\") pod \"11101453-8b37-40af-bdbc-76db842bb667\" (UID: \"11101453-8b37-40af-bdbc-76db842bb667\") "
May 26 16:26:41.039707 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.039681    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11101453-8b37-40af-bdbc-76db842bb667-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "11101453-8b37-40af-bdbc-76db842bb667" (UID: "11101453-8b37-40af-bdbc-76db842bb667"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:26:41.041355 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.041332    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11101453-8b37-40af-bdbc-76db842bb667-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "11101453-8b37-40af-bdbc-76db842bb667" (UID: "11101453-8b37-40af-bdbc-76db842bb667"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:26:41.140277 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.140207    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11101453-8b37-40af-bdbc-76db842bb667-proxy-tls\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:26:41.140277 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.140235    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11101453-8b37-40af-bdbc-76db842bb667-openshift-service-ca-bundle\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:26:41.243212 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.243179    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"]
May 26 16:26:41.243698 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.243449    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6" podUID="13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0" containerName="ensemble-graph-007a5" containerID="cri-o://c7d7d449bb60ff724142f00fc09e2ea8b59cac120c7d6d916e92b301326c225f" gracePeriod=30
May 26 16:26:41.661227 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.661194    2560 generic.go:358] "Generic (PLEG): container finished" podID="11101453-8b37-40af-bdbc-76db842bb667" containerID="415c17be8d1594c4c6dff4babbea68112452f480949ec2d6bcf2d1ee5c4be422" exitCode=0
May 26 16:26:41.661431 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.661235    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l" event={"ID":"11101453-8b37-40af-bdbc-76db842bb667","Type":"ContainerDied","Data":"415c17be8d1594c4c6dff4babbea68112452f480949ec2d6bcf2d1ee5c4be422"}
May 26 16:26:41.661431 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.661255    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"
May 26 16:26:41.661431 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.661268    2560 scope.go:117] "RemoveContainer" containerID="415c17be8d1594c4c6dff4babbea68112452f480949ec2d6bcf2d1ee5c4be422"
May 26 16:26:41.661431 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.661259    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l" event={"ID":"11101453-8b37-40af-bdbc-76db842bb667","Type":"ContainerDied","Data":"f70fd7bff4be2af4504301397b2f049f4ef5f2bcd2a87b0fae7725b0aecfd0ec"}
May 26 16:26:41.669773 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.669701    2560 scope.go:117] "RemoveContainer" containerID="415c17be8d1594c4c6dff4babbea68112452f480949ec2d6bcf2d1ee5c4be422"
May 26 16:26:41.671982 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:26:41.671953    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415c17be8d1594c4c6dff4babbea68112452f480949ec2d6bcf2d1ee5c4be422\": container with ID starting with 415c17be8d1594c4c6dff4babbea68112452f480949ec2d6bcf2d1ee5c4be422 not found: ID does not exist" containerID="415c17be8d1594c4c6dff4babbea68112452f480949ec2d6bcf2d1ee5c4be422"
May 26 16:26:41.672104 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.671993    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415c17be8d1594c4c6dff4babbea68112452f480949ec2d6bcf2d1ee5c4be422"} err="failed to get container status \"415c17be8d1594c4c6dff4babbea68112452f480949ec2d6bcf2d1ee5c4be422\": rpc error: code = NotFound desc = could not find container \"415c17be8d1594c4c6dff4babbea68112452f480949ec2d6bcf2d1ee5c4be422\": container with ID starting with 415c17be8d1594c4c6dff4babbea68112452f480949ec2d6bcf2d1ee5c4be422 not found: ID does not exist"
May 26 16:26:41.680002 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.679980    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"]
May 26 16:26:41.683019 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:41.682998    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ad62d-6db94b859c-qz87l"]
May 26 16:26:43.170503 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:43.170470    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11101453-8b37-40af-bdbc-76db842bb667" path="/var/lib/kubelet/pods/11101453-8b37-40af-bdbc-76db842bb667/volumes"
May 26 16:26:43.642588 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:43.642547    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6" podUID="13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0" containerName="ensemble-graph-007a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:26:48.642834 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:48.642783    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6" podUID="13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0" containerName="ensemble-graph-007a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:26:53.641646 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:53.641604    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6" podUID="13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0" containerName="ensemble-graph-007a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:26:53.642118 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:53.641712    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"
May 26 16:26:58.642611 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:26:58.642569    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6" podUID="13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0" containerName="ensemble-graph-007a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:27:03.641759 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:03.641719    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6" podUID="13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0" containerName="ensemble-graph-007a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:27:08.641780 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:08.641742    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6" podUID="13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0" containerName="ensemble-graph-007a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:27:10.960284 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:10.960251    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"]
May 26 16:27:10.960858 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:10.960838    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11101453-8b37-40af-bdbc-76db842bb667" containerName="sequence-graph-ad62d"
May 26 16:27:10.960941 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:10.960860    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="11101453-8b37-40af-bdbc-76db842bb667" containerName="sequence-graph-ad62d"
May 26 16:27:10.960995 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:10.960946    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="11101453-8b37-40af-bdbc-76db842bb667" containerName="sequence-graph-ad62d"
May 26 16:27:10.965260 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:10.965239    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"
May 26 16:27:10.967491 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:10.967459    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-a007a-kube-rbac-proxy-sar-config\""
May 26 16:27:10.967491 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:10.967483    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-a007a-serving-cert\""
May 26 16:27:10.970730 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:10.970710    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"]
May 26 16:27:11.095406 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.095365    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5e0edc4-b648-4966-8d61-5e288e8ec0b8-openshift-service-ca-bundle\") pod \"sequence-graph-a007a-5bd657f6bf-98m9p\" (UID: \"a5e0edc4-b648-4966-8d61-5e288e8ec0b8\") " pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"
May 26 16:27:11.095595 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.095438    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5e0edc4-b648-4966-8d61-5e288e8ec0b8-proxy-tls\") pod \"sequence-graph-a007a-5bd657f6bf-98m9p\" (UID: \"a5e0edc4-b648-4966-8d61-5e288e8ec0b8\") " pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"
May 26 16:27:11.196871 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.196840    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5e0edc4-b648-4966-8d61-5e288e8ec0b8-proxy-tls\") pod \"sequence-graph-a007a-5bd657f6bf-98m9p\" (UID: \"a5e0edc4-b648-4966-8d61-5e288e8ec0b8\") " pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"
May 26 16:27:11.197051 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.196945    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5e0edc4-b648-4966-8d61-5e288e8ec0b8-openshift-service-ca-bundle\") pod \"sequence-graph-a007a-5bd657f6bf-98m9p\" (UID: \"a5e0edc4-b648-4966-8d61-5e288e8ec0b8\") " pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"
May 26 16:27:11.197051 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:27:11.196995    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-a007a-serving-cert: secret "sequence-graph-a007a-serving-cert" not found
May 26 16:27:11.197134 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:27:11.197076    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5e0edc4-b648-4966-8d61-5e288e8ec0b8-proxy-tls podName:a5e0edc4-b648-4966-8d61-5e288e8ec0b8 nodeName:}" failed. No retries permitted until 2026-05-26 16:27:11.697054827 +0000 UTC m=+1549.120647981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a5e0edc4-b648-4966-8d61-5e288e8ec0b8-proxy-tls") pod "sequence-graph-a007a-5bd657f6bf-98m9p" (UID: "a5e0edc4-b648-4966-8d61-5e288e8ec0b8") : secret "sequence-graph-a007a-serving-cert" not found
May 26 16:27:11.197679 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.197660    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5e0edc4-b648-4966-8d61-5e288e8ec0b8-openshift-service-ca-bundle\") pod \"sequence-graph-a007a-5bd657f6bf-98m9p\" (UID: \"a5e0edc4-b648-4966-8d61-5e288e8ec0b8\") " pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"
May 26 16:27:11.390349 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.390329    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"
May 26 16:27:11.499662 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.499579    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0-openshift-service-ca-bundle\") pod \"13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0\" (UID: \"13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0\") "
May 26 16:27:11.499662 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.499621    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0-proxy-tls\") pod \"13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0\" (UID: \"13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0\") "
May 26 16:27:11.499928 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.499905    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0" (UID: "13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:27:11.501691 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.501672    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0" (UID: "13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:27:11.600418 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.600358    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0-openshift-service-ca-bundle\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:27:11.600418 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.600416    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0-proxy-tls\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:27:11.701232 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.701200    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5e0edc4-b648-4966-8d61-5e288e8ec0b8-proxy-tls\") pod \"sequence-graph-a007a-5bd657f6bf-98m9p\" (UID: \"a5e0edc4-b648-4966-8d61-5e288e8ec0b8\") " pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"
May 26 16:27:11.703659 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.703635    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5e0edc4-b648-4966-8d61-5e288e8ec0b8-proxy-tls\") pod \"sequence-graph-a007a-5bd657f6bf-98m9p\" (UID: \"a5e0edc4-b648-4966-8d61-5e288e8ec0b8\") " pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"
May 26 16:27:11.747873 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.747842    2560 generic.go:358] "Generic (PLEG): container finished" podID="13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0" containerID="c7d7d449bb60ff724142f00fc09e2ea8b59cac120c7d6d916e92b301326c225f" exitCode=0
May 26 16:27:11.748031 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.747884    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6" event={"ID":"13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0","Type":"ContainerDied","Data":"c7d7d449bb60ff724142f00fc09e2ea8b59cac120c7d6d916e92b301326c225f"}
May 26 16:27:11.748031 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.747906    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"
May 26 16:27:11.748031 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.747918    2560 scope.go:117] "RemoveContainer" containerID="c7d7d449bb60ff724142f00fc09e2ea8b59cac120c7d6d916e92b301326c225f"
May 26 16:27:11.748031 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.747906    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6" event={"ID":"13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0","Type":"ContainerDied","Data":"8431f562df9b8f2b4aab96f807d0d252bd795fb0f15d1d1478e91141a513ce29"}
May 26 16:27:11.756934 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.756916    2560 scope.go:117] "RemoveContainer" containerID="c7d7d449bb60ff724142f00fc09e2ea8b59cac120c7d6d916e92b301326c225f"
May 26 16:27:11.757174 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:27:11.757156    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d7d449bb60ff724142f00fc09e2ea8b59cac120c7d6d916e92b301326c225f\": container with ID starting with c7d7d449bb60ff724142f00fc09e2ea8b59cac120c7d6d916e92b301326c225f not found: ID does not exist" containerID="c7d7d449bb60ff724142f00fc09e2ea8b59cac120c7d6d916e92b301326c225f"
May 26 16:27:11.757221 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.757182    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d7d449bb60ff724142f00fc09e2ea8b59cac120c7d6d916e92b301326c225f"} err="failed to get container status \"c7d7d449bb60ff724142f00fc09e2ea8b59cac120c7d6d916e92b301326c225f\": rpc error: code = NotFound desc = could not find container \"c7d7d449bb60ff724142f00fc09e2ea8b59cac120c7d6d916e92b301326c225f\": container with ID starting with c7d7d449bb60ff724142f00fc09e2ea8b59cac120c7d6d916e92b301326c225f not found: ID does not exist"
May 26 16:27:11.770435 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.770375    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"]
May 26 16:27:11.774164 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.774145    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-007a5-866fd95dbf-w2kn6"]
May 26 16:27:11.876768 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:11.876740    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"
May 26 16:27:12.001751 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:12.001616    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"]
May 26 16:27:12.004412 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:27:12.004367    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e0edc4_b648_4966_8d61_5e288e8ec0b8.slice/crio-db2494adf1d3f61fd60d88f6504707e4c8b42a4f5c76bf180a1cbcc5d9afd57e WatchSource:0}: Error finding container db2494adf1d3f61fd60d88f6504707e4c8b42a4f5c76bf180a1cbcc5d9afd57e: Status 404 returned error can't find the container with id db2494adf1d3f61fd60d88f6504707e4c8b42a4f5c76bf180a1cbcc5d9afd57e
May 26 16:27:12.751778 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:12.751737    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p" event={"ID":"a5e0edc4-b648-4966-8d61-5e288e8ec0b8","Type":"ContainerStarted","Data":"6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29"}
May 26 16:27:12.751778 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:12.751782    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p" event={"ID":"a5e0edc4-b648-4966-8d61-5e288e8ec0b8","Type":"ContainerStarted","Data":"db2494adf1d3f61fd60d88f6504707e4c8b42a4f5c76bf180a1cbcc5d9afd57e"}
May 26 16:27:12.752024 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:12.751986    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"
May 26 16:27:12.769096 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:12.769047    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p" podStartSLOduration=2.7690336220000002 podStartE2EDuration="2.769033622s" podCreationTimestamp="2026-05-26 16:27:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:27:12.767837811 +0000 UTC m=+1550.191430973" watchObservedRunningTime="2026-05-26 16:27:12.769033622 +0000 UTC m=+1550.192626788"
May 26 16:27:13.170488 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:13.170375    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0" path="/var/lib/kubelet/pods/13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0/volumes"
May 26 16:27:18.761849 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:18.761814    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"
May 26 16:27:21.029948 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:21.029909    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"]
May 26 16:27:21.030324 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:21.030223    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p" podUID="a5e0edc4-b648-4966-8d61-5e288e8ec0b8" containerName="sequence-graph-a007a" containerID="cri-o://6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29" gracePeriod=30
May 26 16:27:23.760120 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:23.760078    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p" podUID="a5e0edc4-b648-4966-8d61-5e288e8ec0b8" containerName="sequence-graph-a007a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:27:28.760245 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:28.760203    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p" podUID="a5e0edc4-b648-4966-8d61-5e288e8ec0b8" containerName="sequence-graph-a007a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:27:33.759821 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:33.759772    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p" podUID="a5e0edc4-b648-4966-8d61-5e288e8ec0b8" containerName="sequence-graph-a007a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:27:33.760284 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:33.759873    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"
May 26 16:27:38.760497 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:38.760460    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p" podUID="a5e0edc4-b648-4966-8d61-5e288e8ec0b8" containerName="sequence-graph-a007a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:27:41.469701 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:41.469663    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"]
May 26 16:27:41.470166 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:41.470122    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0" containerName="ensemble-graph-007a5"
May 26 16:27:41.470166 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:41.470141    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0" containerName="ensemble-graph-007a5"
May 26 16:27:41.470279 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:41.470248    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="13f28c0b-3a9c-4fc3-b1f9-67f11f2be0a0" containerName="ensemble-graph-007a5"
May 26 16:27:41.474485 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:41.474465    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"
May 26 16:27:41.476828 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:41.476806    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-2138e-serving-cert\""
May 26 16:27:41.477040 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:41.477023    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-2138e-kube-rbac-proxy-sar-config\""
May 26 16:27:41.480917 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:41.480897    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"]
May 26 16:27:41.533015 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:41.532984    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe976bc8-2285-426e-9c70-ff4c7e869605-proxy-tls\") pod \"ensemble-graph-2138e-94f4bbc69-5zxlj\" (UID: \"fe976bc8-2285-426e-9c70-ff4c7e869605\") " pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"
May 26 16:27:41.533185 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:41.533036    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe976bc8-2285-426e-9c70-ff4c7e869605-openshift-service-ca-bundle\") pod \"ensemble-graph-2138e-94f4bbc69-5zxlj\" (UID: \"fe976bc8-2285-426e-9c70-ff4c7e869605\") " pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"
May 26 16:27:41.633655 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:41.633627    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe976bc8-2285-426e-9c70-ff4c7e869605-openshift-service-ca-bundle\") pod \"ensemble-graph-2138e-94f4bbc69-5zxlj\" (UID: \"fe976bc8-2285-426e-9c70-ff4c7e869605\") " pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"
May 26 16:27:41.633845 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:41.633708    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe976bc8-2285-426e-9c70-ff4c7e869605-proxy-tls\") pod \"ensemble-graph-2138e-94f4bbc69-5zxlj\" (UID: \"fe976bc8-2285-426e-9c70-ff4c7e869605\") " pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"
May 26 16:27:41.633922 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:27:41.633851    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-2138e-serving-cert: secret "ensemble-graph-2138e-serving-cert" not found
May 26 16:27:41.633975 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:27:41.633946    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe976bc8-2285-426e-9c70-ff4c7e869605-proxy-tls podName:fe976bc8-2285-426e-9c70-ff4c7e869605 nodeName:}" failed. No retries permitted until 2026-05-26 16:27:42.13392098 +0000 UTC m=+1579.557514135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fe976bc8-2285-426e-9c70-ff4c7e869605-proxy-tls") pod "ensemble-graph-2138e-94f4bbc69-5zxlj" (UID: "fe976bc8-2285-426e-9c70-ff4c7e869605") : secret "ensemble-graph-2138e-serving-cert" not found
May 26 16:27:41.634258 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:41.634237    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe976bc8-2285-426e-9c70-ff4c7e869605-openshift-service-ca-bundle\") pod \"ensemble-graph-2138e-94f4bbc69-5zxlj\" (UID: \"fe976bc8-2285-426e-9c70-ff4c7e869605\") " pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"
May 26 16:27:42.137173 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:42.137137    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe976bc8-2285-426e-9c70-ff4c7e869605-proxy-tls\") pod \"ensemble-graph-2138e-94f4bbc69-5zxlj\" (UID: \"fe976bc8-2285-426e-9c70-ff4c7e869605\") " pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"
May 26 16:27:42.139520 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:42.139502    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe976bc8-2285-426e-9c70-ff4c7e869605-proxy-tls\") pod \"ensemble-graph-2138e-94f4bbc69-5zxlj\" (UID: \"fe976bc8-2285-426e-9c70-ff4c7e869605\") " pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"
May 26 16:27:42.385515 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:42.385470    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"
May 26 16:27:42.511163 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:42.511127    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"]
May 26 16:27:42.843669 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:42.843637    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj" event={"ID":"fe976bc8-2285-426e-9c70-ff4c7e869605","Type":"ContainerStarted","Data":"1a5d4956e465e7dd5fc0f6e8aedf8139bb842114a0da663a05443822d313fbfe"}
May 26 16:27:42.843669 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:42.843670    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj" event={"ID":"fe976bc8-2285-426e-9c70-ff4c7e869605","Type":"ContainerStarted","Data":"28786cd685def5d6a0a75cf6dc1a99488b21b0c16efdd564a2d01c1d25b15ed5"}
May 26 16:27:42.843881 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:42.843694    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"
May 26 16:27:42.863643 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:42.863596    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj" podStartSLOduration=1.863582388 podStartE2EDuration="1.863582388s" podCreationTimestamp="2026-05-26 16:27:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:27:42.861476815 +0000 UTC m=+1580.285069976" watchObservedRunningTime="2026-05-26 16:27:42.863582388 +0000 UTC m=+1580.287175574"
May 26 16:27:43.760596 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:43.760554    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p" podUID="a5e0edc4-b648-4966-8d61-5e288e8ec0b8" containerName="sequence-graph-a007a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:27:48.759935 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:48.759900    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p" podUID="a5e0edc4-b648-4966-8d61-5e288e8ec0b8" containerName="sequence-graph-a007a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:27:48.852687 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:48.852659    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"
May 26 16:27:51.054065 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:27:51.054031    2560 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e0edc4_b648_4966_8d61_5e288e8ec0b8.slice/crio-db2494adf1d3f61fd60d88f6504707e4c8b42a4f5c76bf180a1cbcc5d9afd57e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e0edc4_b648_4966_8d61_5e288e8ec0b8.slice/crio-conmon-6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29.scope\": RecentStats: unable to find data in memory cache]"
May 26 16:27:51.054458 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:27:51.054093    2560 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e0edc4_b648_4966_8d61_5e288e8ec0b8.slice/crio-6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29.scope\": RecentStats: unable to find data in memory cache]"
May 26 16:27:51.054458 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:27:51.054314    2560 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e0edc4_b648_4966_8d61_5e288e8ec0b8.slice/crio-conmon-6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e0edc4_b648_4966_8d61_5e288e8ec0b8.slice/crio-6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29.scope\": RecentStats: unable to find data in memory cache]"
May 26 16:27:51.184007 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:51.183978    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"
May 26 16:27:51.318044 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:51.317951    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5e0edc4-b648-4966-8d61-5e288e8ec0b8-openshift-service-ca-bundle\") pod \"a5e0edc4-b648-4966-8d61-5e288e8ec0b8\" (UID: \"a5e0edc4-b648-4966-8d61-5e288e8ec0b8\") "
May 26 16:27:51.318044 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:51.317993    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5e0edc4-b648-4966-8d61-5e288e8ec0b8-proxy-tls\") pod \"a5e0edc4-b648-4966-8d61-5e288e8ec0b8\" (UID: \"a5e0edc4-b648-4966-8d61-5e288e8ec0b8\") "
May 26 16:27:51.318331 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:51.318307    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e0edc4-b648-4966-8d61-5e288e8ec0b8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a5e0edc4-b648-4966-8d61-5e288e8ec0b8" (UID: "a5e0edc4-b648-4966-8d61-5e288e8ec0b8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:27:51.320052 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:51.320027    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e0edc4-b648-4966-8d61-5e288e8ec0b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a5e0edc4-b648-4966-8d61-5e288e8ec0b8" (UID: "a5e0edc4-b648-4966-8d61-5e288e8ec0b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:27:51.418707 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:51.418673    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5e0edc4-b648-4966-8d61-5e288e8ec0b8-openshift-service-ca-bundle\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:27:51.418707 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:51.418704    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5e0edc4-b648-4966-8d61-5e288e8ec0b8-proxy-tls\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:27:51.867845 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:51.867804    2560 generic.go:358] "Generic (PLEG): container finished" podID="a5e0edc4-b648-4966-8d61-5e288e8ec0b8" containerID="6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29" exitCode=0
May 26 16:27:51.868014 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:51.867868    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"
May 26 16:27:51.868014 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:51.867891    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p" event={"ID":"a5e0edc4-b648-4966-8d61-5e288e8ec0b8","Type":"ContainerDied","Data":"6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29"}
May 26 16:27:51.868014 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:51.867928    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p" event={"ID":"a5e0edc4-b648-4966-8d61-5e288e8ec0b8","Type":"ContainerDied","Data":"db2494adf1d3f61fd60d88f6504707e4c8b42a4f5c76bf180a1cbcc5d9afd57e"}
May 26 16:27:51.868014 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:51.867943    2560 scope.go:117] "RemoveContainer" containerID="6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29"
May 26 16:27:51.876454 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:51.876438    2560 scope.go:117] "RemoveContainer" containerID="6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29"
May 26 16:27:51.876689 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:27:51.876672    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29\": container with ID starting with 6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29 not found: ID does not exist" containerID="6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29"
May 26 16:27:51.876752 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:51.876702    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29"} err="failed to get container status \"6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29\": rpc error: code = NotFound desc = could not find container \"6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29\": container with ID starting with 6e08c3a4b5f19cdfdc494b19ae7e0a01deb3f05a072b0522f58a6e024b4a0d29 not found: ID does not exist"
May 26 16:27:51.889243 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:51.889217    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"]
May 26 16:27:51.892693 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:51.892673    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a007a-5bd657f6bf-98m9p"]
May 26 16:27:53.171029 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:27:53.170993    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5e0edc4-b648-4966-8d61-5e288e8ec0b8" path="/var/lib/kubelet/pods/a5e0edc4-b648-4966-8d61-5e288e8ec0b8/volumes"
May 26 16:28:21.247111 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:21.247030    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"]
May 26 16:28:21.247687 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:21.247371    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5e0edc4-b648-4966-8d61-5e288e8ec0b8" containerName="sequence-graph-a007a"
May 26 16:28:21.247687 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:21.247384    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5e0edc4-b648-4966-8d61-5e288e8ec0b8" containerName="sequence-graph-a007a"
May 26 16:28:21.247687 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:21.247529    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="a5e0edc4-b648-4966-8d61-5e288e8ec0b8" containerName="sequence-graph-a007a"
May 26 16:28:21.250472 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:21.250457    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"
May 26 16:28:21.252627 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:21.252605    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-86373-serving-cert\""
May 26 16:28:21.252855 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:21.252842    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-86373-kube-rbac-proxy-sar-config\""
May 26 16:28:21.260488 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:21.260463    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"]
May 26 16:28:21.260812 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:21.260785    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac-openshift-service-ca-bundle\") pod \"sequence-graph-86373-6d887b9df8-c9zkb\" (UID: \"a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac\") " pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"
May 26 16:28:21.260915 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:21.260882    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac-proxy-tls\") pod \"sequence-graph-86373-6d887b9df8-c9zkb\" (UID: \"a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac\") " pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"
May 26 16:28:21.361240 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:21.361207    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac-proxy-tls\") pod \"sequence-graph-86373-6d887b9df8-c9zkb\" (UID: \"a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac\") " pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"
May 26 16:28:21.361423 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:21.361260    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac-openshift-service-ca-bundle\") pod \"sequence-graph-86373-6d887b9df8-c9zkb\" (UID: \"a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac\") " pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"
May 26 16:28:21.361423 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:28:21.361379    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-86373-serving-cert: secret "sequence-graph-86373-serving-cert" not found
May 26 16:28:21.361508 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:28:21.361466    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac-proxy-tls podName:a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac nodeName:}" failed. No retries permitted until 2026-05-26 16:28:21.861447597 +0000 UTC m=+1619.285040741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac-proxy-tls") pod "sequence-graph-86373-6d887b9df8-c9zkb" (UID: "a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac") : secret "sequence-graph-86373-serving-cert" not found
May 26 16:28:21.361906 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:21.361888    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac-openshift-service-ca-bundle\") pod \"sequence-graph-86373-6d887b9df8-c9zkb\" (UID: \"a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac\") " pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"
May 26 16:28:21.865760 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:21.865721    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac-proxy-tls\") pod \"sequence-graph-86373-6d887b9df8-c9zkb\" (UID: \"a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac\") " pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"
May 26 16:28:21.868123 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:21.868096    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac-proxy-tls\") pod \"sequence-graph-86373-6d887b9df8-c9zkb\" (UID: \"a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac\") " pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"
May 26 16:28:22.161800 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:22.161715    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"
May 26 16:28:22.283864 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:22.283828    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"]
May 26 16:28:22.287851 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:28:22.287826    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1df1e87_7cd1_4f46_8782_ba4dd1bd73ac.slice/crio-678b89853fd909a7fd722e8cc8d8312bf3c23c157c515da1b3f82dee16b7b742 WatchSource:0}: Error finding container 678b89853fd909a7fd722e8cc8d8312bf3c23c157c515da1b3f82dee16b7b742: Status 404 returned error can't find the container with id 678b89853fd909a7fd722e8cc8d8312bf3c23c157c515da1b3f82dee16b7b742
May 26 16:28:22.962760 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:22.962720    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb" event={"ID":"a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac","Type":"ContainerStarted","Data":"45a76cdd03e00b04860590e40156530556621b7854d62ff79e13c58c08a8ee2f"}
May 26 16:28:22.962760 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:22.962761    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb" event={"ID":"a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac","Type":"ContainerStarted","Data":"678b89853fd909a7fd722e8cc8d8312bf3c23c157c515da1b3f82dee16b7b742"}
May 26 16:28:22.963010 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:22.962849    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"
May 26 16:28:22.982498 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:22.982429    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb" podStartSLOduration=1.982413277 podStartE2EDuration="1.982413277s" podCreationTimestamp="2026-05-26 16:28:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:28:22.981167383 +0000 UTC m=+1620.404760544" watchObservedRunningTime="2026-05-26 16:28:22.982413277 +0000 UTC m=+1620.406006439"
May 26 16:28:28.972141 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:28:28.972108    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"
May 26 16:31:23.144194 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:31:23.144168    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:31:23.146805 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:31:23.146783    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:35:56.084697 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:35:56.084661    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"]
May 26 16:35:56.087041 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:35:56.084894    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj" podUID="fe976bc8-2285-426e-9c70-ff4c7e869605" containerName="ensemble-graph-2138e" containerID="cri-o://1a5d4956e465e7dd5fc0f6e8aedf8139bb842114a0da663a05443822d313fbfe" gracePeriod=30
May 26 16:35:58.850757 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:35:58.850712    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj" podUID="fe976bc8-2285-426e-9c70-ff4c7e869605" containerName="ensemble-graph-2138e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:36:03.850085 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:03.850045    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj" podUID="fe976bc8-2285-426e-9c70-ff4c7e869605" containerName="ensemble-graph-2138e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:36:08.850218 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:08.850170    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj" podUID="fe976bc8-2285-426e-9c70-ff4c7e869605" containerName="ensemble-graph-2138e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:36:08.850665 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:08.850269    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"
May 26 16:36:13.850882 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:13.850837    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj" podUID="fe976bc8-2285-426e-9c70-ff4c7e869605" containerName="ensemble-graph-2138e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:36:18.850271 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:18.850227    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj" podUID="fe976bc8-2285-426e-9c70-ff4c7e869605" containerName="ensemble-graph-2138e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:36:23.165896 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:23.165866    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:36:23.169767 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:23.169743    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:36:23.850509 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:23.850471    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj" podUID="fe976bc8-2285-426e-9c70-ff4c7e869605" containerName="ensemble-graph-2138e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:36:26.221650 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:26.221624    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"
May 26 16:36:26.372025 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:26.371941    2560 generic.go:358] "Generic (PLEG): container finished" podID="fe976bc8-2285-426e-9c70-ff4c7e869605" containerID="1a5d4956e465e7dd5fc0f6e8aedf8139bb842114a0da663a05443822d313fbfe" exitCode=0
May 26 16:36:26.372025 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:26.372011    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"
May 26 16:36:26.372202 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:26.372028    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj" event={"ID":"fe976bc8-2285-426e-9c70-ff4c7e869605","Type":"ContainerDied","Data":"1a5d4956e465e7dd5fc0f6e8aedf8139bb842114a0da663a05443822d313fbfe"}
May 26 16:36:26.372202 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:26.372069    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj" event={"ID":"fe976bc8-2285-426e-9c70-ff4c7e869605","Type":"ContainerDied","Data":"28786cd685def5d6a0a75cf6dc1a99488b21b0c16efdd564a2d01c1d25b15ed5"}
May 26 16:36:26.372202 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:26.372084    2560 scope.go:117] "RemoveContainer" containerID="1a5d4956e465e7dd5fc0f6e8aedf8139bb842114a0da663a05443822d313fbfe"
May 26 16:36:26.379713 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:26.379697    2560 scope.go:117] "RemoveContainer" containerID="1a5d4956e465e7dd5fc0f6e8aedf8139bb842114a0da663a05443822d313fbfe"
May 26 16:36:26.379945 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:36:26.379928    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a5d4956e465e7dd5fc0f6e8aedf8139bb842114a0da663a05443822d313fbfe\": container with ID starting with 1a5d4956e465e7dd5fc0f6e8aedf8139bb842114a0da663a05443822d313fbfe not found: ID does not exist" containerID="1a5d4956e465e7dd5fc0f6e8aedf8139bb842114a0da663a05443822d313fbfe"
May 26 16:36:26.380001 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:26.379954    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a5d4956e465e7dd5fc0f6e8aedf8139bb842114a0da663a05443822d313fbfe"} err="failed to get container status \"1a5d4956e465e7dd5fc0f6e8aedf8139bb842114a0da663a05443822d313fbfe\": rpc error: code = NotFound desc = could not find container \"1a5d4956e465e7dd5fc0f6e8aedf8139bb842114a0da663a05443822d313fbfe\": container with ID starting with 1a5d4956e465e7dd5fc0f6e8aedf8139bb842114a0da663a05443822d313fbfe not found: ID does not exist"
May 26 16:36:26.382167 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:26.382152    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe976bc8-2285-426e-9c70-ff4c7e869605-proxy-tls\") pod \"fe976bc8-2285-426e-9c70-ff4c7e869605\" (UID: \"fe976bc8-2285-426e-9c70-ff4c7e869605\") "
May 26 16:36:26.382228 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:26.382212    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe976bc8-2285-426e-9c70-ff4c7e869605-openshift-service-ca-bundle\") pod \"fe976bc8-2285-426e-9c70-ff4c7e869605\" (UID: \"fe976bc8-2285-426e-9c70-ff4c7e869605\") "
May 26 16:36:26.382501 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:26.382479    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe976bc8-2285-426e-9c70-ff4c7e869605-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "fe976bc8-2285-426e-9c70-ff4c7e869605" (UID: "fe976bc8-2285-426e-9c70-ff4c7e869605"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:36:26.384018 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:26.383995    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe976bc8-2285-426e-9c70-ff4c7e869605-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe976bc8-2285-426e-9c70-ff4c7e869605" (UID: "fe976bc8-2285-426e-9c70-ff4c7e869605"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:36:26.483159 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:26.483129    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe976bc8-2285-426e-9c70-ff4c7e869605-proxy-tls\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:36:26.483159 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:26.483157    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe976bc8-2285-426e-9c70-ff4c7e869605-openshift-service-ca-bundle\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:36:26.698996 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:26.698966    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"]
May 26 16:36:26.703421 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:26.703383    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-2138e-94f4bbc69-5zxlj"]
May 26 16:36:27.170764 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:27.170734    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe976bc8-2285-426e-9c70-ff4c7e869605" path="/var/lib/kubelet/pods/fe976bc8-2285-426e-9c70-ff4c7e869605/volumes"
May 26 16:36:35.915677 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:35.915648    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"]
May 26 16:36:35.916080 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:35.915869    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb" podUID="a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac" containerName="sequence-graph-86373" containerID="cri-o://45a76cdd03e00b04860590e40156530556621b7854d62ff79e13c58c08a8ee2f" gracePeriod=30
May 26 16:36:38.970196 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:38.970157    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb" podUID="a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac" containerName="sequence-graph-86373" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:36:43.969449 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:43.969385    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb" podUID="a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac" containerName="sequence-graph-86373" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:36:48.969645 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:48.969606    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb" podUID="a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac" containerName="sequence-graph-86373" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:36:48.969992 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:48.969719    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"
May 26 16:36:53.969619 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:53.969574    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb" podUID="a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac" containerName="sequence-graph-86373" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:36:56.443353 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:56.443320    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"]
May 26 16:36:56.443808 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:56.443790    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe976bc8-2285-426e-9c70-ff4c7e869605" containerName="ensemble-graph-2138e"
May 26 16:36:56.443850 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:56.443812    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe976bc8-2285-426e-9c70-ff4c7e869605" containerName="ensemble-graph-2138e"
May 26 16:36:56.443909 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:56.443899    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="fe976bc8-2285-426e-9c70-ff4c7e869605" containerName="ensemble-graph-2138e"
May 26 16:36:56.446955 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:56.446940    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"
May 26 16:36:56.449579 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:56.449554    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-82fbc-serving-cert\""
May 26 16:36:56.449687 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:56.449602    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-82fbc-kube-rbac-proxy-sar-config\""
May 26 16:36:56.462067 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:56.462045    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"]
May 26 16:36:56.507052 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:56.507024    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/533c8237-ca5f-421b-8004-0c57d78ace35-openshift-service-ca-bundle\") pod \"splitter-graph-82fbc-dbf78cc7b-2vw2x\" (UID: \"533c8237-ca5f-421b-8004-0c57d78ace35\") " pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"
May 26 16:36:56.507180 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:56.507072    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/533c8237-ca5f-421b-8004-0c57d78ace35-proxy-tls\") pod \"splitter-graph-82fbc-dbf78cc7b-2vw2x\" (UID: \"533c8237-ca5f-421b-8004-0c57d78ace35\") " pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"
May 26 16:36:56.608381 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:56.608345    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/533c8237-ca5f-421b-8004-0c57d78ace35-openshift-service-ca-bundle\") pod \"splitter-graph-82fbc-dbf78cc7b-2vw2x\" (UID: \"533c8237-ca5f-421b-8004-0c57d78ace35\") " pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"
May 26 16:36:56.608546 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:56.608427    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/533c8237-ca5f-421b-8004-0c57d78ace35-proxy-tls\") pod \"splitter-graph-82fbc-dbf78cc7b-2vw2x\" (UID: \"533c8237-ca5f-421b-8004-0c57d78ace35\") " pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"
May 26 16:36:56.608611 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:36:56.608553    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-82fbc-serving-cert: secret "splitter-graph-82fbc-serving-cert" not found
May 26 16:36:56.608683 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:36:56.608617    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533c8237-ca5f-421b-8004-0c57d78ace35-proxy-tls podName:533c8237-ca5f-421b-8004-0c57d78ace35 nodeName:}" failed. No retries permitted until 2026-05-26 16:36:57.108595645 +0000 UTC m=+2134.532188791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/533c8237-ca5f-421b-8004-0c57d78ace35-proxy-tls") pod "splitter-graph-82fbc-dbf78cc7b-2vw2x" (UID: "533c8237-ca5f-421b-8004-0c57d78ace35") : secret "splitter-graph-82fbc-serving-cert" not found
May 26 16:36:56.609050 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:56.609024    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/533c8237-ca5f-421b-8004-0c57d78ace35-openshift-service-ca-bundle\") pod \"splitter-graph-82fbc-dbf78cc7b-2vw2x\" (UID: \"533c8237-ca5f-421b-8004-0c57d78ace35\") " pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"
May 26 16:36:57.112194 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:57.112158    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/533c8237-ca5f-421b-8004-0c57d78ace35-proxy-tls\") pod \"splitter-graph-82fbc-dbf78cc7b-2vw2x\" (UID: \"533c8237-ca5f-421b-8004-0c57d78ace35\") " pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"
May 26 16:36:57.114548 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:57.114531    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/533c8237-ca5f-421b-8004-0c57d78ace35-proxy-tls\") pod \"splitter-graph-82fbc-dbf78cc7b-2vw2x\" (UID: \"533c8237-ca5f-421b-8004-0c57d78ace35\") " pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"
May 26 16:36:57.357195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:57.357162    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"
May 26 16:36:57.482331 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:57.482307    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"]
May 26 16:36:57.484762 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:36:57.484732    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533c8237_ca5f_421b_8004_0c57d78ace35.slice/crio-2a7fbb70c23bbf36e44dc0ee44be2663a19472dfb2264c7ff989e71a1849e20a WatchSource:0}: Error finding container 2a7fbb70c23bbf36e44dc0ee44be2663a19472dfb2264c7ff989e71a1849e20a: Status 404 returned error can't find the container with id 2a7fbb70c23bbf36e44dc0ee44be2663a19472dfb2264c7ff989e71a1849e20a
May 26 16:36:57.486432 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:57.486412    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:36:58.465096 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:58.465058    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x" event={"ID":"533c8237-ca5f-421b-8004-0c57d78ace35","Type":"ContainerStarted","Data":"943f1e34e2cf55c88ab1ec4c3eb62033b876b6291fc380a73ddd3afaa57cc5de"}
May 26 16:36:58.465096 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:58.465095    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x" event={"ID":"533c8237-ca5f-421b-8004-0c57d78ace35","Type":"ContainerStarted","Data":"2a7fbb70c23bbf36e44dc0ee44be2663a19472dfb2264c7ff989e71a1849e20a"}
May 26 16:36:58.465302 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:58.465194    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"
May 26 16:36:58.481590 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:58.481547    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x" podStartSLOduration=2.48153397 podStartE2EDuration="2.48153397s" podCreationTimestamp="2026-05-26 16:36:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:36:58.480699791 +0000 UTC m=+2135.904292952" watchObservedRunningTime="2026-05-26 16:36:58.48153397 +0000 UTC m=+2135.905127131"
May 26 16:36:58.970248 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:36:58.970208    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb" podUID="a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac" containerName="sequence-graph-86373" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:37:03.969800 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:03.969715    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb" podUID="a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac" containerName="sequence-graph-86373" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:37:04.472812 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:04.472782    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"
May 26 16:37:06.057787 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.057762    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"
May 26 16:37:06.186506 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.186425    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac-proxy-tls\") pod \"a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac\" (UID: \"a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac\") "
May 26 16:37:06.186506 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.186468    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac-openshift-service-ca-bundle\") pod \"a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac\" (UID: \"a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac\") "
May 26 16:37:06.186852 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.186826    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac" (UID: "a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:37:06.188358 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.188337    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac" (UID: "a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:37:06.288017 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.287985    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac-proxy-tls\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:37:06.288017 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.288012    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac-openshift-service-ca-bundle\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:37:06.455183 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.455107    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"]
May 26 16:37:06.455365 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.455328    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x" podUID="533c8237-ca5f-421b-8004-0c57d78ace35" containerName="splitter-graph-82fbc" containerID="cri-o://943f1e34e2cf55c88ab1ec4c3eb62033b876b6291fc380a73ddd3afaa57cc5de" gracePeriod=30
May 26 16:37:06.488908 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.488877    2560 generic.go:358] "Generic (PLEG): container finished" podID="a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac" containerID="45a76cdd03e00b04860590e40156530556621b7854d62ff79e13c58c08a8ee2f" exitCode=0
May 26 16:37:06.489054 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.488927    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb" event={"ID":"a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac","Type":"ContainerDied","Data":"45a76cdd03e00b04860590e40156530556621b7854d62ff79e13c58c08a8ee2f"}
May 26 16:37:06.489054 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.488945    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"
May 26 16:37:06.489054 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.488960    2560 scope.go:117] "RemoveContainer" containerID="45a76cdd03e00b04860590e40156530556621b7854d62ff79e13c58c08a8ee2f"
May 26 16:37:06.489054 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.488950    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb" event={"ID":"a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac","Type":"ContainerDied","Data":"678b89853fd909a7fd722e8cc8d8312bf3c23c157c515da1b3f82dee16b7b742"}
May 26 16:37:06.496552 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.496532    2560 scope.go:117] "RemoveContainer" containerID="45a76cdd03e00b04860590e40156530556621b7854d62ff79e13c58c08a8ee2f"
May 26 16:37:06.496858 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:37:06.496836    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a76cdd03e00b04860590e40156530556621b7854d62ff79e13c58c08a8ee2f\": container with ID starting with 45a76cdd03e00b04860590e40156530556621b7854d62ff79e13c58c08a8ee2f not found: ID does not exist" containerID="45a76cdd03e00b04860590e40156530556621b7854d62ff79e13c58c08a8ee2f"
May 26 16:37:06.496950 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.496863    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a76cdd03e00b04860590e40156530556621b7854d62ff79e13c58c08a8ee2f"} err="failed to get container status \"45a76cdd03e00b04860590e40156530556621b7854d62ff79e13c58c08a8ee2f\": rpc error: code = NotFound desc = could not find container \"45a76cdd03e00b04860590e40156530556621b7854d62ff79e13c58c08a8ee2f\": container with ID starting with 45a76cdd03e00b04860590e40156530556621b7854d62ff79e13c58c08a8ee2f not found: ID does not exist"
May 26 16:37:06.511459 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.511430    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"]
May 26 16:37:06.516127 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:06.516106    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-86373-6d887b9df8-c9zkb"]
May 26 16:37:07.171033 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:07.170995    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac" path="/var/lib/kubelet/pods/a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac/volumes"
May 26 16:37:09.471913 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:09.471876    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x" podUID="533c8237-ca5f-421b-8004-0c57d78ace35" containerName="splitter-graph-82fbc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:37:14.471341 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:14.471294    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x" podUID="533c8237-ca5f-421b-8004-0c57d78ace35" containerName="splitter-graph-82fbc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:37:19.471730 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:19.471685    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x" podUID="533c8237-ca5f-421b-8004-0c57d78ace35" containerName="splitter-graph-82fbc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:37:19.472121 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:19.471785    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"
May 26 16:37:24.471697 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:24.471653    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x" podUID="533c8237-ca5f-421b-8004-0c57d78ace35" containerName="splitter-graph-82fbc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:37:29.471346 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:29.471304    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x" podUID="533c8237-ca5f-421b-8004-0c57d78ace35" containerName="splitter-graph-82fbc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:37:34.472288 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:34.472240    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x" podUID="533c8237-ca5f-421b-8004-0c57d78ace35" containerName="splitter-graph-82fbc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:37:36.151773 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.151736    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"]
May 26 16:37:36.152201 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.152065    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac" containerName="sequence-graph-86373"
May 26 16:37:36.152201 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.152077    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac" containerName="sequence-graph-86373"
May 26 16:37:36.152201 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.152131    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="a1df1e87-7cd1-4f46-8782-ba4dd1bd73ac" containerName="sequence-graph-86373"
May 26 16:37:36.156343 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.156325    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"
May 26 16:37:36.158293 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.158273    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bffff-serving-cert\""
May 26 16:37:36.158539 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.158514    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bffff-kube-rbac-proxy-sar-config\""
May 26 16:37:36.169574 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.169551    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"]
May 26 16:37:36.336555 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.336518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/021f32c3-aaff-47f1-a253-83888421c2e6-openshift-service-ca-bundle\") pod \"switch-graph-bffff-7994c5b4b7-cdjp9\" (UID: \"021f32c3-aaff-47f1-a253-83888421c2e6\") " pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"
May 26 16:37:36.336729 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.336561    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/021f32c3-aaff-47f1-a253-83888421c2e6-proxy-tls\") pod \"switch-graph-bffff-7994c5b4b7-cdjp9\" (UID: \"021f32c3-aaff-47f1-a253-83888421c2e6\") " pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"
May 26 16:37:36.437349 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.437265    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/021f32c3-aaff-47f1-a253-83888421c2e6-openshift-service-ca-bundle\") pod \"switch-graph-bffff-7994c5b4b7-cdjp9\" (UID: \"021f32c3-aaff-47f1-a253-83888421c2e6\") " pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"
May 26 16:37:36.437349 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.437308    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/021f32c3-aaff-47f1-a253-83888421c2e6-proxy-tls\") pod \"switch-graph-bffff-7994c5b4b7-cdjp9\" (UID: \"021f32c3-aaff-47f1-a253-83888421c2e6\") " pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"
May 26 16:37:36.437922 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.437901    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/021f32c3-aaff-47f1-a253-83888421c2e6-openshift-service-ca-bundle\") pod \"switch-graph-bffff-7994c5b4b7-cdjp9\" (UID: \"021f32c3-aaff-47f1-a253-83888421c2e6\") " pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"
May 26 16:37:36.439831 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.439805    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/021f32c3-aaff-47f1-a253-83888421c2e6-proxy-tls\") pod \"switch-graph-bffff-7994c5b4b7-cdjp9\" (UID: \"021f32c3-aaff-47f1-a253-83888421c2e6\") " pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"
May 26 16:37:36.466475 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.466453    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"
May 26 16:37:36.530024 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:37:36.529896    2560 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533c8237_ca5f_421b_8004_0c57d78ace35.slice/crio-943f1e34e2cf55c88ab1ec4c3eb62033b876b6291fc380a73ddd3afaa57cc5de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533c8237_ca5f_421b_8004_0c57d78ace35.slice/crio-conmon-943f1e34e2cf55c88ab1ec4c3eb62033b876b6291fc380a73ddd3afaa57cc5de.scope\": RecentStats: unable to find data in memory cache]"
May 26 16:37:36.586535 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.586502    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x" event={"ID":"533c8237-ca5f-421b-8004-0c57d78ace35","Type":"ContainerDied","Data":"943f1e34e2cf55c88ab1ec4c3eb62033b876b6291fc380a73ddd3afaa57cc5de"}
May 26 16:37:36.586709 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.586678    2560 generic.go:358] "Generic (PLEG): container finished" podID="533c8237-ca5f-421b-8004-0c57d78ace35" containerID="943f1e34e2cf55c88ab1ec4c3eb62033b876b6291fc380a73ddd3afaa57cc5de" exitCode=0
May 26 16:37:36.615120 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.615095    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"]
May 26 16:37:36.618124 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:37:36.618077    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021f32c3_aaff_47f1_a253_83888421c2e6.slice/crio-35dd62501c95638ee17863bd71cd5c82d3af49f92c8ee0933bb3e34c7432ad7e WatchSource:0}: Error finding container 35dd62501c95638ee17863bd71cd5c82d3af49f92c8ee0933bb3e34c7432ad7e: Status 404 returned error can't find the container with id 35dd62501c95638ee17863bd71cd5c82d3af49f92c8ee0933bb3e34c7432ad7e
May 26 16:37:36.659633 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.659611    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"
May 26 16:37:36.839156 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.839125    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/533c8237-ca5f-421b-8004-0c57d78ace35-openshift-service-ca-bundle\") pod \"533c8237-ca5f-421b-8004-0c57d78ace35\" (UID: \"533c8237-ca5f-421b-8004-0c57d78ace35\") "
May 26 16:37:36.839156 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.839158    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/533c8237-ca5f-421b-8004-0c57d78ace35-proxy-tls\") pod \"533c8237-ca5f-421b-8004-0c57d78ace35\" (UID: \"533c8237-ca5f-421b-8004-0c57d78ace35\") "
May 26 16:37:36.839594 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.839564    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533c8237-ca5f-421b-8004-0c57d78ace35-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "533c8237-ca5f-421b-8004-0c57d78ace35" (UID: "533c8237-ca5f-421b-8004-0c57d78ace35"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:37:36.841225 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.841209    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533c8237-ca5f-421b-8004-0c57d78ace35-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "533c8237-ca5f-421b-8004-0c57d78ace35" (UID: "533c8237-ca5f-421b-8004-0c57d78ace35"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:37:36.939931 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.939895    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/533c8237-ca5f-421b-8004-0c57d78ace35-openshift-service-ca-bundle\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:37:36.939931 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:36.939924    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/533c8237-ca5f-421b-8004-0c57d78ace35-proxy-tls\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:37:37.590606 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:37.590574    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9" event={"ID":"021f32c3-aaff-47f1-a253-83888421c2e6","Type":"ContainerStarted","Data":"7ce32c94c8660e61251f25a8fa3181ad83c49f8ea4068b7482b96715ee4e3409"}
May 26 16:37:37.591017 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:37.590613    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9" event={"ID":"021f32c3-aaff-47f1-a253-83888421c2e6","Type":"ContainerStarted","Data":"35dd62501c95638ee17863bd71cd5c82d3af49f92c8ee0933bb3e34c7432ad7e"}
May 26 16:37:37.591017 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:37.590658    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"
May 26 16:37:37.591679 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:37.591655    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x" event={"ID":"533c8237-ca5f-421b-8004-0c57d78ace35","Type":"ContainerDied","Data":"2a7fbb70c23bbf36e44dc0ee44be2663a19472dfb2264c7ff989e71a1849e20a"}
May 26 16:37:37.591747 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:37.591700    2560 scope.go:117] "RemoveContainer" containerID="943f1e34e2cf55c88ab1ec4c3eb62033b876b6291fc380a73ddd3afaa57cc5de"
May 26 16:37:37.591785 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:37.591703    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"
May 26 16:37:37.606138 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:37.605980    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9" podStartSLOduration=1.6059642429999998 podStartE2EDuration="1.605964243s" podCreationTimestamp="2026-05-26 16:37:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:37:37.605344271 +0000 UTC m=+2175.028937437" watchObservedRunningTime="2026-05-26 16:37:37.605964243 +0000 UTC m=+2175.029557405"
May 26 16:37:37.617108 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:37.617082    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"]
May 26 16:37:37.619248 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:37.619229    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-82fbc-dbf78cc7b-2vw2x"]
May 26 16:37:39.170930 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:39.170893    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="533c8237-ca5f-421b-8004-0c57d78ace35" path="/var/lib/kubelet/pods/533c8237-ca5f-421b-8004-0c57d78ace35/volumes"
May 26 16:37:43.603864 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:37:43.603836    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"
May 26 16:38:06.669348 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:06.669312    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"]
May 26 16:38:06.669763 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:06.669675    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533c8237-ca5f-421b-8004-0c57d78ace35" containerName="splitter-graph-82fbc"
May 26 16:38:06.669763 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:06.669688    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="533c8237-ca5f-421b-8004-0c57d78ace35" containerName="splitter-graph-82fbc"
May 26 16:38:06.669763 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:06.669735    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="533c8237-ca5f-421b-8004-0c57d78ace35" containerName="splitter-graph-82fbc"
May 26 16:38:06.672747 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:06.672730    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"
May 26 16:38:06.674921 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:06.674898    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-5798a-kube-rbac-proxy-sar-config\""
May 26 16:38:06.674921 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:06.674898    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-5798a-serving-cert\""
May 26 16:38:06.680958 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:06.680937    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"]
May 26 16:38:06.764104 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:06.764069    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d632a147-bbd1-4829-8e87-16f3c80e5185-proxy-tls\") pod \"splitter-graph-5798a-56688b886c-wnnxz\" (UID: \"d632a147-bbd1-4829-8e87-16f3c80e5185\") " pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"
May 26 16:38:06.764306 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:06.764138    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d632a147-bbd1-4829-8e87-16f3c80e5185-openshift-service-ca-bundle\") pod \"splitter-graph-5798a-56688b886c-wnnxz\" (UID: \"d632a147-bbd1-4829-8e87-16f3c80e5185\") " pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"
May 26 16:38:06.864609 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:06.864576    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d632a147-bbd1-4829-8e87-16f3c80e5185-proxy-tls\") pod \"splitter-graph-5798a-56688b886c-wnnxz\" (UID: \"d632a147-bbd1-4829-8e87-16f3c80e5185\") " pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"
May 26 16:38:06.864791 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:06.864626    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d632a147-bbd1-4829-8e87-16f3c80e5185-openshift-service-ca-bundle\") pod \"splitter-graph-5798a-56688b886c-wnnxz\" (UID: \"d632a147-bbd1-4829-8e87-16f3c80e5185\") " pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"
May 26 16:38:06.864791 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:38:06.864742    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-5798a-serving-cert: secret "splitter-graph-5798a-serving-cert" not found
May 26 16:38:06.864905 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:38:06.864833    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d632a147-bbd1-4829-8e87-16f3c80e5185-proxy-tls podName:d632a147-bbd1-4829-8e87-16f3c80e5185 nodeName:}" failed. No retries permitted until 2026-05-26 16:38:07.364809098 +0000 UTC m=+2204.788402253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d632a147-bbd1-4829-8e87-16f3c80e5185-proxy-tls") pod "splitter-graph-5798a-56688b886c-wnnxz" (UID: "d632a147-bbd1-4829-8e87-16f3c80e5185") : secret "splitter-graph-5798a-serving-cert" not found
May 26 16:38:06.865252 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:06.865234    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d632a147-bbd1-4829-8e87-16f3c80e5185-openshift-service-ca-bundle\") pod \"splitter-graph-5798a-56688b886c-wnnxz\" (UID: \"d632a147-bbd1-4829-8e87-16f3c80e5185\") " pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"
May 26 16:38:07.368987 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:07.368955    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d632a147-bbd1-4829-8e87-16f3c80e5185-proxy-tls\") pod \"splitter-graph-5798a-56688b886c-wnnxz\" (UID: \"d632a147-bbd1-4829-8e87-16f3c80e5185\") " pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"
May 26 16:38:07.371402 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:07.371370    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d632a147-bbd1-4829-8e87-16f3c80e5185-proxy-tls\") pod \"splitter-graph-5798a-56688b886c-wnnxz\" (UID: \"d632a147-bbd1-4829-8e87-16f3c80e5185\") " pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"
May 26 16:38:07.583526 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:07.583491    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"
May 26 16:38:07.710894 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:07.710864    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"]
May 26 16:38:07.717091 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:38:07.717060    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd632a147_bbd1_4829_8e87_16f3c80e5185.slice/crio-c39f78d70b26a1a5da8b19d1eb010a13bb4026bdd6d1f058f7e75da802c8f0a9 WatchSource:0}: Error finding container c39f78d70b26a1a5da8b19d1eb010a13bb4026bdd6d1f058f7e75da802c8f0a9: Status 404 returned error can't find the container with id c39f78d70b26a1a5da8b19d1eb010a13bb4026bdd6d1f058f7e75da802c8f0a9
May 26 16:38:08.683044 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:08.683010    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz" event={"ID":"d632a147-bbd1-4829-8e87-16f3c80e5185","Type":"ContainerStarted","Data":"cf810ea644b6cec8798f5f7f0e60e36ffb0261a7271fea862c6c67ee2ab55f6a"}
May 26 16:38:08.683044 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:08.683049    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz" event={"ID":"d632a147-bbd1-4829-8e87-16f3c80e5185","Type":"ContainerStarted","Data":"c39f78d70b26a1a5da8b19d1eb010a13bb4026bdd6d1f058f7e75da802c8f0a9"}
May 26 16:38:08.683252 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:08.683138    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"
May 26 16:38:08.699582 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:08.699543    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz" podStartSLOduration=2.699531682 podStartE2EDuration="2.699531682s" podCreationTimestamp="2026-05-26 16:38:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:38:08.69766309 +0000 UTC m=+2206.121256250" watchObservedRunningTime="2026-05-26 16:38:08.699531682 +0000 UTC m=+2206.123124843"
May 26 16:38:14.690885 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:38:14.690850    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"
May 26 16:41:23.189196 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:41:23.189169    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:41:23.192682 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:41:23.192665    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:46:21.290578 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:21.290538    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"]
May 26 16:46:21.291157 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:21.290841    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz" podUID="d632a147-bbd1-4829-8e87-16f3c80e5185" containerName="splitter-graph-5798a" containerID="cri-o://cf810ea644b6cec8798f5f7f0e60e36ffb0261a7271fea862c6c67ee2ab55f6a" gracePeriod=30
May 26 16:46:23.217009 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:23.216900    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:46:23.222043 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:23.220818    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:46:24.690482 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:24.690437    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz" podUID="d632a147-bbd1-4829-8e87-16f3c80e5185" containerName="splitter-graph-5798a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:46:29.689994 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:29.689954    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz" podUID="d632a147-bbd1-4829-8e87-16f3c80e5185" containerName="splitter-graph-5798a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:46:34.689913 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:34.689874    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz" podUID="d632a147-bbd1-4829-8e87-16f3c80e5185" containerName="splitter-graph-5798a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:46:34.690335 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:34.689981    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"
May 26 16:46:39.689980 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:39.689940    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz" podUID="d632a147-bbd1-4829-8e87-16f3c80e5185" containerName="splitter-graph-5798a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:46:44.689747 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:44.689702    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz" podUID="d632a147-bbd1-4829-8e87-16f3c80e5185" containerName="splitter-graph-5798a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:46:49.689630 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:49.689585    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz" podUID="d632a147-bbd1-4829-8e87-16f3c80e5185" containerName="splitter-graph-5798a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:46:51.432788 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:51.432765    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"
May 26 16:46:51.528725 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:51.528694    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d632a147-bbd1-4829-8e87-16f3c80e5185-proxy-tls\") pod \"d632a147-bbd1-4829-8e87-16f3c80e5185\" (UID: \"d632a147-bbd1-4829-8e87-16f3c80e5185\") "
May 26 16:46:51.528725 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:51.528730    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d632a147-bbd1-4829-8e87-16f3c80e5185-openshift-service-ca-bundle\") pod \"d632a147-bbd1-4829-8e87-16f3c80e5185\" (UID: \"d632a147-bbd1-4829-8e87-16f3c80e5185\") "
May 26 16:46:51.529157 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:51.529124    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d632a147-bbd1-4829-8e87-16f3c80e5185-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d632a147-bbd1-4829-8e87-16f3c80e5185" (UID: "d632a147-bbd1-4829-8e87-16f3c80e5185"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:46:51.530747 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:51.530716    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d632a147-bbd1-4829-8e87-16f3c80e5185-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d632a147-bbd1-4829-8e87-16f3c80e5185" (UID: "d632a147-bbd1-4829-8e87-16f3c80e5185"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:46:51.629521 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:51.629433    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d632a147-bbd1-4829-8e87-16f3c80e5185-proxy-tls\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:46:51.629521 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:51.629466    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d632a147-bbd1-4829-8e87-16f3c80e5185-openshift-service-ca-bundle\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:46:52.211784 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:52.211747    2560 generic.go:358] "Generic (PLEG): container finished" podID="d632a147-bbd1-4829-8e87-16f3c80e5185" containerID="cf810ea644b6cec8798f5f7f0e60e36ffb0261a7271fea862c6c67ee2ab55f6a" exitCode=0
May 26 16:46:52.212043 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:52.211812    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"
May 26 16:46:52.212043 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:52.211831    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz" event={"ID":"d632a147-bbd1-4829-8e87-16f3c80e5185","Type":"ContainerDied","Data":"cf810ea644b6cec8798f5f7f0e60e36ffb0261a7271fea862c6c67ee2ab55f6a"}
May 26 16:46:52.212043 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:52.211875    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz" event={"ID":"d632a147-bbd1-4829-8e87-16f3c80e5185","Type":"ContainerDied","Data":"c39f78d70b26a1a5da8b19d1eb010a13bb4026bdd6d1f058f7e75da802c8f0a9"}
May 26 16:46:52.212043 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:52.211892    2560 scope.go:117] "RemoveContainer" containerID="cf810ea644b6cec8798f5f7f0e60e36ffb0261a7271fea862c6c67ee2ab55f6a"
May 26 16:46:52.220320 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:52.220300    2560 scope.go:117] "RemoveContainer" containerID="cf810ea644b6cec8798f5f7f0e60e36ffb0261a7271fea862c6c67ee2ab55f6a"
May 26 16:46:52.220587 ip-10-0-137-193 kubenswrapper[2560]: E0526 16:46:52.220566    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf810ea644b6cec8798f5f7f0e60e36ffb0261a7271fea862c6c67ee2ab55f6a\": container with ID starting with cf810ea644b6cec8798f5f7f0e60e36ffb0261a7271fea862c6c67ee2ab55f6a not found: ID does not exist" containerID="cf810ea644b6cec8798f5f7f0e60e36ffb0261a7271fea862c6c67ee2ab55f6a"
May 26 16:46:52.220667 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:52.220596    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf810ea644b6cec8798f5f7f0e60e36ffb0261a7271fea862c6c67ee2ab55f6a"} err="failed to get container status \"cf810ea644b6cec8798f5f7f0e60e36ffb0261a7271fea862c6c67ee2ab55f6a\": rpc error: code = NotFound desc = could not find container \"cf810ea644b6cec8798f5f7f0e60e36ffb0261a7271fea862c6c67ee2ab55f6a\": container with ID starting with cf810ea644b6cec8798f5f7f0e60e36ffb0261a7271fea862c6c67ee2ab55f6a not found: ID does not exist"
May 26 16:46:52.232848 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:52.232823    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"]
May 26 16:46:52.236761 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:52.236739    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5798a-56688b886c-wnnxz"]
May 26 16:46:53.170329 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:46:53.170289    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d632a147-bbd1-4829-8e87-16f3c80e5185" path="/var/lib/kubelet/pods/d632a147-bbd1-4829-8e87-16f3c80e5185/volumes"
May 26 16:51:23.238277 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:51:23.238250    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:51:23.242931 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:51:23.242910    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:53:55.362351 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:55.362271    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"]
May 26 16:53:55.362873 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:55.362531    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9" podUID="021f32c3-aaff-47f1-a253-83888421c2e6" containerName="switch-graph-bffff" containerID="cri-o://7ce32c94c8660e61251f25a8fa3181ad83c49f8ea4068b7482b96715ee4e3409" gracePeriod=30
May 26 16:53:57.066813 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.066773    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-56l56/must-gather-gzpfp"]
May 26 16:53:57.067164 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.067082    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d632a147-bbd1-4829-8e87-16f3c80e5185" containerName="splitter-graph-5798a"
May 26 16:53:57.067164 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.067093    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d632a147-bbd1-4829-8e87-16f3c80e5185" containerName="splitter-graph-5798a"
May 26 16:53:57.067164 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.067156    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="d632a147-bbd1-4829-8e87-16f3c80e5185" containerName="splitter-graph-5798a"
May 26 16:53:57.069056 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.069037    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-56l56/must-gather-gzpfp"
May 26 16:53:57.071435 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.071416    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-56l56\"/\"kube-root-ca.crt\""
May 26 16:53:57.072324 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.072304    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-56l56\"/\"openshift-service-ca.crt\""
May 26 16:53:57.072442 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.072306    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-56l56\"/\"default-dockercfg-fwhwq\""
May 26 16:53:57.083325 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.083304    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-56l56/must-gather-gzpfp"]
May 26 16:53:57.094421 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.094382    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d685e06-308e-4fe7-aa40-35c778c13bf8-must-gather-output\") pod \"must-gather-gzpfp\" (UID: \"3d685e06-308e-4fe7-aa40-35c778c13bf8\") " pod="openshift-must-gather-56l56/must-gather-gzpfp"
May 26 16:53:57.094498 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.094442    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcdbk\" (UniqueName: \"kubernetes.io/projected/3d685e06-308e-4fe7-aa40-35c778c13bf8-kube-api-access-mcdbk\") pod \"must-gather-gzpfp\" (UID: \"3d685e06-308e-4fe7-aa40-35c778c13bf8\") " pod="openshift-must-gather-56l56/must-gather-gzpfp"
May 26 16:53:57.195159 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.195130    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d685e06-308e-4fe7-aa40-35c778c13bf8-must-gather-output\") pod \"must-gather-gzpfp\" (UID: \"3d685e06-308e-4fe7-aa40-35c778c13bf8\") " pod="openshift-must-gather-56l56/must-gather-gzpfp"
May 26 16:53:57.195312 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.195170    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mcdbk\" (UniqueName: \"kubernetes.io/projected/3d685e06-308e-4fe7-aa40-35c778c13bf8-kube-api-access-mcdbk\") pod \"must-gather-gzpfp\" (UID: \"3d685e06-308e-4fe7-aa40-35c778c13bf8\") " pod="openshift-must-gather-56l56/must-gather-gzpfp"
May 26 16:53:57.195517 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.195497    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d685e06-308e-4fe7-aa40-35c778c13bf8-must-gather-output\") pod \"must-gather-gzpfp\" (UID: \"3d685e06-308e-4fe7-aa40-35c778c13bf8\") " pod="openshift-must-gather-56l56/must-gather-gzpfp"
May 26 16:53:57.203852 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.203824    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcdbk\" (UniqueName: \"kubernetes.io/projected/3d685e06-308e-4fe7-aa40-35c778c13bf8-kube-api-access-mcdbk\") pod \"must-gather-gzpfp\" (UID: \"3d685e06-308e-4fe7-aa40-35c778c13bf8\") " pod="openshift-must-gather-56l56/must-gather-gzpfp"
May 26 16:53:57.390668 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.390585    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-56l56/must-gather-gzpfp"
May 26 16:53:57.514649 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.514611    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-56l56/must-gather-gzpfp"]
May 26 16:53:57.521661 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:53:57.521634    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d685e06_308e_4fe7_aa40_35c778c13bf8.slice/crio-dcca9e84582d919c6cad19854aaef8dd524d59be4329e436067c3203d0b5e84f WatchSource:0}: Error finding container dcca9e84582d919c6cad19854aaef8dd524d59be4329e436067c3203d0b5e84f: Status 404 returned error can't find the container with id dcca9e84582d919c6cad19854aaef8dd524d59be4329e436067c3203d0b5e84f
May 26 16:53:57.523239 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:57.523223    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:53:58.419431 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:58.419373    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-56l56/must-gather-gzpfp" event={"ID":"3d685e06-308e-4fe7-aa40-35c778c13bf8","Type":"ContainerStarted","Data":"dcca9e84582d919c6cad19854aaef8dd524d59be4329e436067c3203d0b5e84f"}
May 26 16:53:58.605382 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:53:58.605339    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9" podUID="021f32c3-aaff-47f1-a253-83888421c2e6" containerName="switch-graph-bffff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:54:03.603312 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:03.603275    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9" podUID="021f32c3-aaff-47f1-a253-83888421c2e6" containerName="switch-graph-bffff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:54:04.442905 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:04.442865    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-56l56/must-gather-gzpfp" event={"ID":"3d685e06-308e-4fe7-aa40-35c778c13bf8","Type":"ContainerStarted","Data":"6ed707a0092483fbc804ec6f18498704434151487e367d02ea5160c91ddff347"}
May 26 16:54:04.442905 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:04.442914    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-56l56/must-gather-gzpfp" event={"ID":"3d685e06-308e-4fe7-aa40-35c778c13bf8","Type":"ContainerStarted","Data":"9cbd0db1b608b0b09695c237737159a06064047eece799efdbe916ac9fc642f4"}
May 26 16:54:04.460728 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:04.460671    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-56l56/must-gather-gzpfp" podStartSLOduration=1.458007075 podStartE2EDuration="7.460654812s" podCreationTimestamp="2026-05-26 16:53:57 +0000 UTC" firstStartedPulling="2026-05-26 16:53:57.523339819 +0000 UTC m=+3154.946932958" lastFinishedPulling="2026-05-26 16:54:03.525987554 +0000 UTC m=+3160.949580695" observedRunningTime="2026-05-26 16:54:04.4583136 +0000 UTC m=+3161.881906761" watchObservedRunningTime="2026-05-26 16:54:04.460654812 +0000 UTC m=+3161.884247974"
May 26 16:54:08.602240 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:08.602202    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9" podUID="021f32c3-aaff-47f1-a253-83888421c2e6" containerName="switch-graph-bffff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:54:08.602639 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:08.602315    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"
May 26 16:54:12.378956 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:12.378927    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bffff-7994c5b4b7-cdjp9_021f32c3-aaff-47f1-a253-83888421c2e6/switch-graph-bffff/0.log"
May 26 16:54:13.193195 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:13.193171    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bffff-7994c5b4b7-cdjp9_021f32c3-aaff-47f1-a253-83888421c2e6/switch-graph-bffff/0.log"
May 26 16:54:13.604926 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:13.604885    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9" podUID="021f32c3-aaff-47f1-a253-83888421c2e6" containerName="switch-graph-bffff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:54:13.987163 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:13.987138    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bffff-7994c5b4b7-cdjp9_021f32c3-aaff-47f1-a253-83888421c2e6/switch-graph-bffff/0.log"
May 26 16:54:14.813529 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:14.813501    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bffff-7994c5b4b7-cdjp9_021f32c3-aaff-47f1-a253-83888421c2e6/switch-graph-bffff/0.log"
May 26 16:54:15.636571 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:15.636544    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bffff-7994c5b4b7-cdjp9_021f32c3-aaff-47f1-a253-83888421c2e6/switch-graph-bffff/0.log"
May 26 16:54:16.439667 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:16.439635    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bffff-7994c5b4b7-cdjp9_021f32c3-aaff-47f1-a253-83888421c2e6/switch-graph-bffff/0.log"
May 26 16:54:17.219768 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:17.219728    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bffff-7994c5b4b7-cdjp9_021f32c3-aaff-47f1-a253-83888421c2e6/switch-graph-bffff/0.log"
May 26 16:54:18.035283 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:18.035259    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bffff-7994c5b4b7-cdjp9_021f32c3-aaff-47f1-a253-83888421c2e6/switch-graph-bffff/0.log"
May 26 16:54:18.601799 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:18.601758    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9" podUID="021f32c3-aaff-47f1-a253-83888421c2e6" containerName="switch-graph-bffff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:54:18.830519 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:18.830493    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bffff-7994c5b4b7-cdjp9_021f32c3-aaff-47f1-a253-83888421c2e6/switch-graph-bffff/0.log"
May 26 16:54:19.667977 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:19.667927    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bffff-7994c5b4b7-cdjp9_021f32c3-aaff-47f1-a253-83888421c2e6/switch-graph-bffff/0.log"
May 26 16:54:20.478717 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:20.478685    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bffff-7994c5b4b7-cdjp9_021f32c3-aaff-47f1-a253-83888421c2e6/switch-graph-bffff/0.log"
May 26 16:54:21.279801 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:21.279774    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bffff-7994c5b4b7-cdjp9_021f32c3-aaff-47f1-a253-83888421c2e6/switch-graph-bffff/0.log"
May 26 16:54:23.602125 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:23.602087    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9" podUID="021f32c3-aaff-47f1-a253-83888421c2e6" containerName="switch-graph-bffff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:54:25.508772 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:25.508728    2560 generic.go:358] "Generic (PLEG): container finished" podID="021f32c3-aaff-47f1-a253-83888421c2e6" containerID="7ce32c94c8660e61251f25a8fa3181ad83c49f8ea4068b7482b96715ee4e3409" exitCode=0
May 26 16:54:25.509116 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:25.508793    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9" event={"ID":"021f32c3-aaff-47f1-a253-83888421c2e6","Type":"ContainerDied","Data":"7ce32c94c8660e61251f25a8fa3181ad83c49f8ea4068b7482b96715ee4e3409"}
May 26 16:54:26.006118 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:26.006095    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"
May 26 16:54:26.146517 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:26.146479    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/021f32c3-aaff-47f1-a253-83888421c2e6-openshift-service-ca-bundle\") pod \"021f32c3-aaff-47f1-a253-83888421c2e6\" (UID: \"021f32c3-aaff-47f1-a253-83888421c2e6\") "
May 26 16:54:26.146692 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:26.146566    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/021f32c3-aaff-47f1-a253-83888421c2e6-proxy-tls\") pod \"021f32c3-aaff-47f1-a253-83888421c2e6\" (UID: \"021f32c3-aaff-47f1-a253-83888421c2e6\") "
May 26 16:54:26.146822 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:26.146798    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021f32c3-aaff-47f1-a253-83888421c2e6-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "021f32c3-aaff-47f1-a253-83888421c2e6" (UID: "021f32c3-aaff-47f1-a253-83888421c2e6"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:54:26.148417 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:26.148364    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/021f32c3-aaff-47f1-a253-83888421c2e6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "021f32c3-aaff-47f1-a253-83888421c2e6" (UID: "021f32c3-aaff-47f1-a253-83888421c2e6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:54:26.247729 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:26.247645    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/021f32c3-aaff-47f1-a253-83888421c2e6-openshift-service-ca-bundle\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:54:26.247729 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:26.247674    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/021f32c3-aaff-47f1-a253-83888421c2e6-proxy-tls\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:54:26.512613 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:26.512526    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9" event={"ID":"021f32c3-aaff-47f1-a253-83888421c2e6","Type":"ContainerDied","Data":"35dd62501c95638ee17863bd71cd5c82d3af49f92c8ee0933bb3e34c7432ad7e"}
May 26 16:54:26.512613 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:26.512559    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"
May 26 16:54:26.513090 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:26.512569    2560 scope.go:117] "RemoveContainer" containerID="7ce32c94c8660e61251f25a8fa3181ad83c49f8ea4068b7482b96715ee4e3409"
May 26 16:54:26.514226 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:26.514198    2560 generic.go:358] "Generic (PLEG): container finished" podID="3d685e06-308e-4fe7-aa40-35c778c13bf8" containerID="9cbd0db1b608b0b09695c237737159a06064047eece799efdbe916ac9fc642f4" exitCode=0
May 26 16:54:26.514319 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:26.514248    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-56l56/must-gather-gzpfp" event={"ID":"3d685e06-308e-4fe7-aa40-35c778c13bf8","Type":"ContainerDied","Data":"9cbd0db1b608b0b09695c237737159a06064047eece799efdbe916ac9fc642f4"}
May 26 16:54:26.514562 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:26.514549    2560 scope.go:117] "RemoveContainer" containerID="9cbd0db1b608b0b09695c237737159a06064047eece799efdbe916ac9fc642f4"
May 26 16:54:26.549601 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:26.549568    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"]
May 26 16:54:26.558272 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:26.558247    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bffff-7994c5b4b7-cdjp9"]
May 26 16:54:26.733484 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:26.733456    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-56l56_must-gather-gzpfp_3d685e06-308e-4fe7-aa40-35c778c13bf8/gather/0.log"
May 26 16:54:27.170294 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.170202    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="021f32c3-aaff-47f1-a253-83888421c2e6" path="/var/lib/kubelet/pods/021f32c3-aaff-47f1-a253-83888421c2e6/volumes"
May 26 16:54:27.326063 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.326032    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sh5kf/must-gather-pwvd6"]
May 26 16:54:27.326357 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.326343    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="021f32c3-aaff-47f1-a253-83888421c2e6" containerName="switch-graph-bffff"
May 26 16:54:27.326419 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.326358    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="021f32c3-aaff-47f1-a253-83888421c2e6" containerName="switch-graph-bffff"
May 26 16:54:27.326461 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.326428    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="021f32c3-aaff-47f1-a253-83888421c2e6" containerName="switch-graph-bffff"
May 26 16:54:27.329674 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.329655    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sh5kf/must-gather-pwvd6"
May 26 16:54:27.332315 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.332290    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sh5kf\"/\"kube-root-ca.crt\""
May 26 16:54:27.332459 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.332302    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sh5kf\"/\"default-dockercfg-hlrkd\""
May 26 16:54:27.332459 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.332332    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sh5kf\"/\"openshift-service-ca.crt\""
May 26 16:54:27.341506 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.341485    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sh5kf/must-gather-pwvd6"]
May 26 16:54:27.457337 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.457252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6wms\" (UniqueName: \"kubernetes.io/projected/4a97b54f-f8fe-4ac1-8438-380213a4e51a-kube-api-access-j6wms\") pod \"must-gather-pwvd6\" (UID: \"4a97b54f-f8fe-4ac1-8438-380213a4e51a\") " pod="openshift-must-gather-sh5kf/must-gather-pwvd6"
May 26 16:54:27.457509 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.457347    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a97b54f-f8fe-4ac1-8438-380213a4e51a-must-gather-output\") pod \"must-gather-pwvd6\" (UID: \"4a97b54f-f8fe-4ac1-8438-380213a4e51a\") " pod="openshift-must-gather-sh5kf/must-gather-pwvd6"
May 26 16:54:27.558139 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.558104    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j6wms\" (UniqueName: \"kubernetes.io/projected/4a97b54f-f8fe-4ac1-8438-380213a4e51a-kube-api-access-j6wms\") pod \"must-gather-pwvd6\" (UID: \"4a97b54f-f8fe-4ac1-8438-380213a4e51a\") " pod="openshift-must-gather-sh5kf/must-gather-pwvd6"
May 26 16:54:27.558549 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.558159    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a97b54f-f8fe-4ac1-8438-380213a4e51a-must-gather-output\") pod \"must-gather-pwvd6\" (UID: \"4a97b54f-f8fe-4ac1-8438-380213a4e51a\") " pod="openshift-must-gather-sh5kf/must-gather-pwvd6"
May 26 16:54:27.558549 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.558529    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a97b54f-f8fe-4ac1-8438-380213a4e51a-must-gather-output\") pod \"must-gather-pwvd6\" (UID: \"4a97b54f-f8fe-4ac1-8438-380213a4e51a\") " pod="openshift-must-gather-sh5kf/must-gather-pwvd6"
May 26 16:54:27.566923 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.566902    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6wms\" (UniqueName: \"kubernetes.io/projected/4a97b54f-f8fe-4ac1-8438-380213a4e51a-kube-api-access-j6wms\") pod \"must-gather-pwvd6\" (UID: \"4a97b54f-f8fe-4ac1-8438-380213a4e51a\") " pod="openshift-must-gather-sh5kf/must-gather-pwvd6"
May 26 16:54:27.638412 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.638370    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sh5kf/must-gather-pwvd6"
May 26 16:54:27.758203 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:27.758169    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sh5kf/must-gather-pwvd6"]
May 26 16:54:27.761153 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:54:27.761121    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a97b54f_f8fe_4ac1_8438_380213a4e51a.slice/crio-8a8fd7146b2d2ed4c05a6850a1e3b38d188ade56ff763c781cdac491c052c056 WatchSource:0}: Error finding container 8a8fd7146b2d2ed4c05a6850a1e3b38d188ade56ff763c781cdac491c052c056: Status 404 returned error can't find the container with id 8a8fd7146b2d2ed4c05a6850a1e3b38d188ade56ff763c781cdac491c052c056
May 26 16:54:28.525800 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:28.525761    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sh5kf/must-gather-pwvd6" event={"ID":"4a97b54f-f8fe-4ac1-8438-380213a4e51a","Type":"ContainerStarted","Data":"8a8fd7146b2d2ed4c05a6850a1e3b38d188ade56ff763c781cdac491c052c056"}
May 26 16:54:29.531143 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:29.531098    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sh5kf/must-gather-pwvd6" event={"ID":"4a97b54f-f8fe-4ac1-8438-380213a4e51a","Type":"ContainerStarted","Data":"041e226b375b3a44f4d36ec3f35076d41912be603befd39becc79d2c40803fde"}
May 26 16:54:29.531143 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:29.531144    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sh5kf/must-gather-pwvd6" event={"ID":"4a97b54f-f8fe-4ac1-8438-380213a4e51a","Type":"ContainerStarted","Data":"32657420badb1f1b3618d9c9963785aa77a6f850e8991c5917070c37f976b142"}
May 26 16:54:29.557125 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:29.557050    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-sh5kf/must-gather-pwvd6" podStartSLOduration=1.65150179 podStartE2EDuration="2.55703414s" podCreationTimestamp="2026-05-26 16:54:27 +0000 UTC" firstStartedPulling="2026-05-26 16:54:27.762939586 +0000 UTC m=+3185.186532726" lastFinishedPulling="2026-05-26 16:54:28.668471934 +0000 UTC m=+3186.092065076" observedRunningTime="2026-05-26 16:54:29.55603175 +0000 UTC m=+3186.979624912" watchObservedRunningTime="2026-05-26 16:54:29.55703414 +0000 UTC m=+3186.980627300"
May 26 16:54:30.182746 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:30.182721    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c7gdw_915dcfd8-ae6f-41be-9694-0f9eb624df9e/global-pull-secret-syncer/0.log"
May 26 16:54:30.355313 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:30.355282    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t2rzl_aaf93059-5a36-41c9-901e-fb7ae0e4224f/konnectivity-agent/0.log"
May 26 16:54:30.426470 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:30.426432    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-193.ec2.internal_e207af1d280d5e1f7a4ab2b460263c14/haproxy/0.log"
May 26 16:54:32.168707 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:32.168666    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-56l56/must-gather-gzpfp"]
May 26 16:54:32.169452 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:32.169375    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-56l56/must-gather-gzpfp" podUID="3d685e06-308e-4fe7-aa40-35c778c13bf8" containerName="copy" containerID="cri-o://6ed707a0092483fbc804ec6f18498704434151487e367d02ea5160c91ddff347" gracePeriod=2
May 26 16:54:32.175749 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:32.175696    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-56l56/must-gather-gzpfp"]
May 26 16:54:32.551831 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:32.551744    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-56l56_must-gather-gzpfp_3d685e06-308e-4fe7-aa40-35c778c13bf8/copy/0.log"
May 26 16:54:32.555615 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:32.555079    2560 generic.go:358] "Generic (PLEG): container finished" podID="3d685e06-308e-4fe7-aa40-35c778c13bf8" containerID="6ed707a0092483fbc804ec6f18498704434151487e367d02ea5160c91ddff347" exitCode=143
May 26 16:54:32.592413 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:32.591629    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-56l56_must-gather-gzpfp_3d685e06-308e-4fe7-aa40-35c778c13bf8/copy/0.log"
May 26 16:54:32.592413 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:32.592106    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-56l56/must-gather-gzpfp"
May 26 16:54:32.596421 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:32.594581    2560 status_manager.go:895] "Failed to get status for pod" podUID="3d685e06-308e-4fe7-aa40-35c778c13bf8" pod="openshift-must-gather-56l56/must-gather-gzpfp" err="pods \"must-gather-gzpfp\" is forbidden: User \"system:node:ip-10-0-137-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-56l56\": no relationship found between node 'ip-10-0-137-193.ec2.internal' and this object"
May 26 16:54:32.709622 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:32.707450    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcdbk\" (UniqueName: \"kubernetes.io/projected/3d685e06-308e-4fe7-aa40-35c778c13bf8-kube-api-access-mcdbk\") pod \"3d685e06-308e-4fe7-aa40-35c778c13bf8\" (UID: \"3d685e06-308e-4fe7-aa40-35c778c13bf8\") "
May 26 16:54:32.709622 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:32.707535    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d685e06-308e-4fe7-aa40-35c778c13bf8-must-gather-output\") pod \"3d685e06-308e-4fe7-aa40-35c778c13bf8\" (UID: \"3d685e06-308e-4fe7-aa40-35c778c13bf8\") "
May 26 16:54:32.709622 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:32.709246    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d685e06-308e-4fe7-aa40-35c778c13bf8-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "3d685e06-308e-4fe7-aa40-35c778c13bf8" (UID: "3d685e06-308e-4fe7-aa40-35c778c13bf8"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:54:32.713459 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:32.713418    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d685e06-308e-4fe7-aa40-35c778c13bf8-kube-api-access-mcdbk" (OuterVolumeSpecName: "kube-api-access-mcdbk") pod "3d685e06-308e-4fe7-aa40-35c778c13bf8" (UID: "3d685e06-308e-4fe7-aa40-35c778c13bf8"). InnerVolumeSpecName "kube-api-access-mcdbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:54:32.808640 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:32.808570    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcdbk\" (UniqueName: \"kubernetes.io/projected/3d685e06-308e-4fe7-aa40-35c778c13bf8-kube-api-access-mcdbk\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:54:32.808640 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:32.808610    2560 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d685e06-308e-4fe7-aa40-35c778c13bf8-must-gather-output\") on node \"ip-10-0-137-193.ec2.internal\" DevicePath \"\""
May 26 16:54:33.172689 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:33.172652    2560 status_manager.go:895] "Failed to get status for pod" podUID="3d685e06-308e-4fe7-aa40-35c778c13bf8" pod="openshift-must-gather-56l56/must-gather-gzpfp" err="pods \"must-gather-gzpfp\" is forbidden: User \"system:node:ip-10-0-137-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-56l56\": no relationship found between node 'ip-10-0-137-193.ec2.internal' and this object"
May 26 16:54:33.184567 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:33.180988    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d685e06-308e-4fe7-aa40-35c778c13bf8" path="/var/lib/kubelet/pods/3d685e06-308e-4fe7-aa40-35c778c13bf8/volumes"
May 26 16:54:33.560429 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:33.560332    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-56l56_must-gather-gzpfp_3d685e06-308e-4fe7-aa40-35c778c13bf8/copy/0.log"
May 26 16:54:33.561737 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:33.560961    2560 scope.go:117] "RemoveContainer" containerID="6ed707a0092483fbc804ec6f18498704434151487e367d02ea5160c91ddff347"
May 26 16:54:33.561737 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:33.561111    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-56l56/must-gather-gzpfp"
May 26 16:54:33.575051 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:33.574736    2560 scope.go:117] "RemoveContainer" containerID="9cbd0db1b608b0b09695c237737159a06064047eece799efdbe916ac9fc642f4"
May 26 16:54:33.926866 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:33.926760    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6c75959455-9zl7z_7028a76e-6db4-4b2c-afd5-d7ab066dc10a/cluster-monitoring-operator/0.log"
May 26 16:54:34.108479 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.108432    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fw8nm_0379f1a0-131d-447a-b24b-ad2c7e39bc50/node-exporter/0.log"
May 26 16:54:34.137219 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.137168    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fw8nm_0379f1a0-131d-447a-b24b-ad2c7e39bc50/kube-rbac-proxy/0.log"
May 26 16:54:34.171591 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.171564    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fw8nm_0379f1a0-131d-447a-b24b-ad2c7e39bc50/init-textfile/0.log"
May 26 16:54:34.368173 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.368139    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-mfjl9_43fccc63-8224-4fd6-8e3a-ef817181d328/kube-rbac-proxy-main/0.log"
May 26 16:54:34.391729 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.391703    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-mfjl9_43fccc63-8224-4fd6-8e3a-ef817181d328/kube-rbac-proxy-self/0.log"
May 26 16:54:34.414689 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.414664    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-mfjl9_43fccc63-8224-4fd6-8e3a-ef817181d328/openshift-state-metrics/0.log"
May 26 16:54:34.456150 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.456107    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2/prometheus/0.log"
May 26 16:54:34.476574 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.476543    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2/config-reloader/0.log"
May 26 16:54:34.506254 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.506216    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2/thanos-sidecar/0.log"
May 26 16:54:34.537528 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.537503    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2/kube-rbac-proxy-web/0.log"
May 26 16:54:34.563796 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.563771    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2/kube-rbac-proxy/0.log"
May 26 16:54:34.593607 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.593579    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2/kube-rbac-proxy-thanos/0.log"
May 26 16:54:34.617786 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.617744    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ca69fa1-9ef6-4e4e-9018-bbc8e11f4ec2/init-config-reloader/0.log"
May 26 16:54:34.655478 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.655384    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-zbhlt_1839dcf6-a640-4f0c-9bc1-e0258415c54b/prometheus-operator/0.log"
May 26 16:54:34.675043 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.674950    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-zbhlt_1839dcf6-a640-4f0c-9bc1-e0258415c54b/kube-rbac-proxy/0.log"
May 26 16:54:34.704846 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.704777    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-vwxs9_533d5f01-445b-4e8e-a3be-7acc89343863/prometheus-operator-admission-webhook/0.log"
May 26 16:54:34.828985 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.828956    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66b5d866b4-7j58t_a3dec2f0-7164-4ef4-a38b-9231faa02de5/thanos-query/0.log"
May 26 16:54:34.851434 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.851382    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66b5d866b4-7j58t_a3dec2f0-7164-4ef4-a38b-9231faa02de5/kube-rbac-proxy-web/0.log"
May 26 16:54:34.878186 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.878155    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66b5d866b4-7j58t_a3dec2f0-7164-4ef4-a38b-9231faa02de5/kube-rbac-proxy/0.log"
May 26 16:54:34.907182 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.907084    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66b5d866b4-7j58t_a3dec2f0-7164-4ef4-a38b-9231faa02de5/prom-label-proxy/0.log"
May 26 16:54:34.940953 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.940922    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66b5d866b4-7j58t_a3dec2f0-7164-4ef4-a38b-9231faa02de5/kube-rbac-proxy-rules/0.log"
May 26 16:54:34.977807 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:34.977783    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66b5d866b4-7j58t_a3dec2f0-7164-4ef4-a38b-9231faa02de5/kube-rbac-proxy-metrics/0.log"
May 26 16:54:36.578351 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:36.578314    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/1.log"
May 26 16:54:36.583360 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:36.583340    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vkhrc_8ad1242b-3f03-48c6-a0cb-f029836842ff/console-operator/2.log"
May 26 16:54:37.316632 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.316601    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"]
May 26 16:54:37.316948 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.316934    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d685e06-308e-4fe7-aa40-35c778c13bf8" containerName="copy"
May 26 16:54:37.316995 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.316954    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d685e06-308e-4fe7-aa40-35c778c13bf8" containerName="copy"
May 26 16:54:37.316995 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.316983    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d685e06-308e-4fe7-aa40-35c778c13bf8" containerName="gather"
May 26 16:54:37.316995 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.316990    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d685e06-308e-4fe7-aa40-35c778c13bf8" containerName="gather"
May 26 16:54:37.317086 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.317068    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="3d685e06-308e-4fe7-aa40-35c778c13bf8" containerName="gather"
May 26 16:54:37.317086 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.317076    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="3d685e06-308e-4fe7-aa40-35c778c13bf8" containerName="copy"
May 26 16:54:37.322175 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.322143    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.331096 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.331050    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"]
May 26 16:54:37.393495 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.393464    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-j57x7_7d3526bf-6f30-4cd2-b6a9-44a0dfc32777/volume-data-source-validator/0.log"
May 26 16:54:37.466208 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.466162    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7b85df15-e1db-4eeb-8eb8-399acefb222b-proc\") pod \"perf-node-gather-daemonset-qqb5z\" (UID: \"7b85df15-e1db-4eeb-8eb8-399acefb222b\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.466448 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.466223    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b85df15-e1db-4eeb-8eb8-399acefb222b-sys\") pod \"perf-node-gather-daemonset-qqb5z\" (UID: \"7b85df15-e1db-4eeb-8eb8-399acefb222b\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.466448 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.466273    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nxjh\" (UniqueName: \"kubernetes.io/projected/7b85df15-e1db-4eeb-8eb8-399acefb222b-kube-api-access-5nxjh\") pod \"perf-node-gather-daemonset-qqb5z\" (UID: \"7b85df15-e1db-4eeb-8eb8-399acefb222b\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.466448 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.466311    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b85df15-e1db-4eeb-8eb8-399acefb222b-lib-modules\") pod \"perf-node-gather-daemonset-qqb5z\" (UID: \"7b85df15-e1db-4eeb-8eb8-399acefb222b\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.466448 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.466381    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7b85df15-e1db-4eeb-8eb8-399acefb222b-podres\") pod \"perf-node-gather-daemonset-qqb5z\" (UID: \"7b85df15-e1db-4eeb-8eb8-399acefb222b\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.567380 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.567302    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b85df15-e1db-4eeb-8eb8-399acefb222b-lib-modules\") pod \"perf-node-gather-daemonset-qqb5z\" (UID: \"7b85df15-e1db-4eeb-8eb8-399acefb222b\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.567551 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.567417    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7b85df15-e1db-4eeb-8eb8-399acefb222b-podres\") pod \"perf-node-gather-daemonset-qqb5z\" (UID: \"7b85df15-e1db-4eeb-8eb8-399acefb222b\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.567551 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.567457    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7b85df15-e1db-4eeb-8eb8-399acefb222b-proc\") pod \"perf-node-gather-daemonset-qqb5z\" (UID: \"7b85df15-e1db-4eeb-8eb8-399acefb222b\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.567551 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.567475    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b85df15-e1db-4eeb-8eb8-399acefb222b-sys\") pod \"perf-node-gather-daemonset-qqb5z\" (UID: \"7b85df15-e1db-4eeb-8eb8-399acefb222b\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.567551 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.567508    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5nxjh\" (UniqueName: \"kubernetes.io/projected/7b85df15-e1db-4eeb-8eb8-399acefb222b-kube-api-access-5nxjh\") pod \"perf-node-gather-daemonset-qqb5z\" (UID: \"7b85df15-e1db-4eeb-8eb8-399acefb222b\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.567551 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.567518    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b85df15-e1db-4eeb-8eb8-399acefb222b-lib-modules\") pod \"perf-node-gather-daemonset-qqb5z\" (UID: \"7b85df15-e1db-4eeb-8eb8-399acefb222b\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.567551 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.567542    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7b85df15-e1db-4eeb-8eb8-399acefb222b-proc\") pod \"perf-node-gather-daemonset-qqb5z\" (UID: \"7b85df15-e1db-4eeb-8eb8-399acefb222b\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.567823 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.567580    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7b85df15-e1db-4eeb-8eb8-399acefb222b-podres\") pod \"perf-node-gather-daemonset-qqb5z\" (UID: \"7b85df15-e1db-4eeb-8eb8-399acefb222b\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.567823 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.567600    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b85df15-e1db-4eeb-8eb8-399acefb222b-sys\") pod \"perf-node-gather-daemonset-qqb5z\" (UID: \"7b85df15-e1db-4eeb-8eb8-399acefb222b\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.577111 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.576979    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nxjh\" (UniqueName: \"kubernetes.io/projected/7b85df15-e1db-4eeb-8eb8-399acefb222b-kube-api-access-5nxjh\") pod \"perf-node-gather-daemonset-qqb5z\" (UID: \"7b85df15-e1db-4eeb-8eb8-399acefb222b\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.634863 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.634817    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:37.785328 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:37.785300    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"]
May 26 16:54:37.787963 ip-10-0-137-193 kubenswrapper[2560]: W0526 16:54:37.787933    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7b85df15_e1db_4eeb_8eb8_399acefb222b.slice/crio-1af115be638a9f6aa1ac424fc06303924328d7089a7186b145052e6f5270014c WatchSource:0}: Error finding container 1af115be638a9f6aa1ac424fc06303924328d7089a7186b145052e6f5270014c: Status 404 returned error can't find the container with id 1af115be638a9f6aa1ac424fc06303924328d7089a7186b145052e6f5270014c
May 26 16:54:38.168449 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:38.168360    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v7sq4_b7627daa-7d0e-4a02-9b00-a3515e6b7137/dns/0.log"
May 26 16:54:38.191114 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:38.191084    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v7sq4_b7627daa-7d0e-4a02-9b00-a3515e6b7137/kube-rbac-proxy/0.log"
May 26 16:54:38.279625 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:38.279598    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xx9vq_a1e36e32-7353-4703-8b16-848a49a47e91/dns-node-resolver/0.log"
May 26 16:54:38.581357 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:38.581323    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z" event={"ID":"7b85df15-e1db-4eeb-8eb8-399acefb222b","Type":"ContainerStarted","Data":"5fbdb1738e72c7c49788db2ef26c0434485d1053869d82e3caf0ee480df4dd58"}
May 26 16:54:38.581357 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:38.581361    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z" event={"ID":"7b85df15-e1db-4eeb-8eb8-399acefb222b","Type":"ContainerStarted","Data":"1af115be638a9f6aa1ac424fc06303924328d7089a7186b145052e6f5270014c"}
May 26 16:54:38.581586 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:38.581410    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:38.597684 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:38.597636    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z" podStartSLOduration=1.597621148 podStartE2EDuration="1.597621148s" podCreationTimestamp="2026-05-26 16:54:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:54:38.595653091 +0000 UTC m=+3196.019246256" watchObservedRunningTime="2026-05-26 16:54:38.597621148 +0000 UTC m=+3196.021214309"
May 26 16:54:38.768403 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:38.768353    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mz8tq_ea5226b3-f441-403d-aa44-8ac33afd311b/node-ca/0.log"
May 26 16:54:39.834290 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:39.834261    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-29s87_fcbf8b71-6b16-4832-b58b-537691709f83/serve-healthcheck-canary/0.log"
May 26 16:54:40.433760 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:40.433733    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ql8x6_517f1a5d-7656-4456-846f-87eb3f4885e6/kube-rbac-proxy/0.log"
May 26 16:54:40.454087 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:40.454067    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ql8x6_517f1a5d-7656-4456-846f-87eb3f4885e6/exporter/0.log"
May 26 16:54:40.477795 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:40.477775    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ql8x6_517f1a5d-7656-4456-846f-87eb3f4885e6/extractor/0.log"
May 26 16:54:42.891531 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:42.891501    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-c4j75_d9ba0862-a9f0-4c34-866c-d45e2ceea18c/s3-init/0.log"
May 26 16:54:44.595481 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:44.595449    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-qqb5z"
May 26 16:54:46.877476 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:46.877342    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-fjj8k_f4b93dc5-3aee-4241-80be-623c0af940da/migrator/0.log"
May 26 16:54:46.899359 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:46.899334    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-fjj8k_f4b93dc5-3aee-4241-80be-623c0af940da/graceful-termination/0.log"
May 26 16:54:47.285782 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:47.285748    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-mmd9t_8f4f1c38-12f6-4b6c-9360-0ba547d5f28a/kube-storage-version-migrator-operator/1.log"
May 26 16:54:47.287066 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:47.287035    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-mmd9t_8f4f1c38-12f6-4b6c-9360-0ba547d5f28a/kube-storage-version-migrator-operator/0.log"
May 26 16:54:48.401406 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:48.401313    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vw726_1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6/kube-multus-additional-cni-plugins/0.log"
May 26 16:54:48.425305 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:48.425278    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vw726_1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6/egress-router-binary-copy/0.log"
May 26 16:54:48.450017 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:48.449981    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vw726_1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6/cni-plugins/0.log"
May 26 16:54:48.473620 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:48.473586    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vw726_1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6/bond-cni-plugin/0.log"
May 26 16:54:48.496467 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:48.496439    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vw726_1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6/routeoverride-cni/0.log"
May 26 16:54:48.521889 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:48.521866    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vw726_1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6/whereabouts-cni-bincopy/0.log"
May 26 16:54:48.544028 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:48.544003    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vw726_1556a0fe-3c5b-4bcc-97e3-f5c361a44bd6/whereabouts-cni/0.log"
May 26 16:54:48.804877 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:48.804849    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rg5xr_0f99d273-701a-4186-836a-333aa040304b/kube-multus/0.log"
May 26 16:54:48.830789 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:48.830764    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-j528d_22219f09-c007-46a5-b425-df729845bf45/network-metrics-daemon/0.log"
May 26 16:54:48.856249 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:48.856229    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-j528d_22219f09-c007-46a5-b425-df729845bf45/kube-rbac-proxy/0.log"
May 26 16:54:50.027508 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:50.027478    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n84fp_078ebf89-62fd-47e8-8a9f-03bf7431c882/ovn-controller/0.log"
May 26 16:54:50.061915 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:50.061894    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n84fp_078ebf89-62fd-47e8-8a9f-03bf7431c882/ovn-acl-logging/0.log"
May 26 16:54:50.091260 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:50.091237    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n84fp_078ebf89-62fd-47e8-8a9f-03bf7431c882/kube-rbac-proxy-node/0.log"
May 26 16:54:50.113058 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:50.113043    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n84fp_078ebf89-62fd-47e8-8a9f-03bf7431c882/kube-rbac-proxy-ovn-metrics/0.log"
May 26 16:54:50.135865 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:50.135850    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n84fp_078ebf89-62fd-47e8-8a9f-03bf7431c882/northd/0.log"
May 26 16:54:50.161275 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:50.161258    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n84fp_078ebf89-62fd-47e8-8a9f-03bf7431c882/nbdb/0.log"
May 26 16:54:50.197883 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:50.197860    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n84fp_078ebf89-62fd-47e8-8a9f-03bf7431c882/sbdb/0.log"
May 26 16:54:50.311317 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:50.311243    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n84fp_078ebf89-62fd-47e8-8a9f-03bf7431c882/ovnkube-controller/0.log"
May 26 16:54:51.660222 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:51.660199    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-dd7q4_67ccb020-7afc-463c-a316-263722c743ad/network-check-target-container/0.log"
May 26 16:54:52.569151 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:52.569120    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-nnztf_1920789f-1573-4118-a9a4-7e3a44fcb1ad/iptables-alerter/0.log"
May 26 16:54:53.240435 ip-10-0-137-193 kubenswrapper[2560]: I0526 16:54:53.240407    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-zgpnk_fc50ed1b-7010-4636-b832-6fd7c64b57da/tuned/0.log"