May 26 11:21:57.053842 ip-10-0-135-71 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 26 11:21:57.053858 ip-10-0-135-71 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 26 11:21:57.053868 ip-10-0-135-71 systemd[1]: kubelet.service: Failed with result 'resources'.
May 26 11:21:57.054221 ip-10-0-135-71 systemd[1]: Failed to start Kubernetes Kubelet.
May 26 11:22:07.243342 ip-10-0-135-71 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 26 11:22:07.243363 ip-10-0-135-71 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 8a29ff84e8a14407b3e249939e313cc3 --
May 26 11:24:27.063694 ip-10-0-135-71 systemd[1]: Starting Kubernetes Kubelet...
May 26 11:24:27.535823 ip-10-0-135-71 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:24:27.535823 ip-10-0-135-71 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 26 11:24:27.535823 ip-10-0-135-71 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:24:27.535823 ip-10-0-135-71 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 26 11:24:27.535823 ip-10-0-135-71 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:24:27.538939 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.538788    2563 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 26 11:24:27.543796 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543779    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:27.543796 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543795    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543799    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543803    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543806    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543809    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543812    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543815    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543818    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543821    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543824    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543826    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543829    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543832    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543834    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543837    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543839    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543842    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543845    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543850    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543853    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:27.543898 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543856    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543858    2563 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543861    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543863    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543866    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543868    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543871    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543874    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543876    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543880    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543884    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543887    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543890    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543894    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543898    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543902    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543905    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543908    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543911    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:27.544405 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543914    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543917    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543920    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543923    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543926    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543929    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543933    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543936    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543938    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543941    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543944    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543946    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543949    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543951    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543954    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543956    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543959    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543962    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543964    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543967    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:27.544863 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543969    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543972    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543975    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543977    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543979    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543982    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543985    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543990    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543992    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543995    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.543998    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544000    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544003    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544005    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544008    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544011    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544013    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544016    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544018    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544021    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:27.545364 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544023    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544027    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544030    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544033    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544036    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544038    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544484    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544490    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544494    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544497    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544500    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544503    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544506    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544509    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544512    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544515    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544517    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544520    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544523    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544526    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:27.545846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544529    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544532    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544535    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544538    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544541    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544543    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544546    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544548    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544551    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544553    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544555    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544558    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544561    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544563    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544566    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544569    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544572    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544574    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544577    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544579    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:27.546344 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544581    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544584    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544587    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544589    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544592    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544594    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544597    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544599    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544601    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544604    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544608    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544611    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544614    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544616    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544619    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544622    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544624    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544627    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544629    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544632    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:27.546912 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544634    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544637    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544639    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544642    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544644    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544646    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544649    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544651    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544654    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544658    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544661    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544663    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544666    2563 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544668    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544671    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544674    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544679    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544681    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544684    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:27.547426 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544686    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544689    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544691    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544694    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544697    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544700    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544703    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544706    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544708    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544711    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544713    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544716    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.544718    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545312    2563 flags.go:64] FLAG: --address="0.0.0.0"
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545322    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545329    2563 flags.go:64] FLAG: --anonymous-auth="true"
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545333    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545338    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545341    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545346    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545350    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 26 11:24:27.547910 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545354    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545357    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545360    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545363    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545367    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545370    2563 flags.go:64] FLAG: --cgroup-root=""
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545373    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545376    2563 flags.go:64] FLAG: --client-ca-file=""
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545379    2563 flags.go:64] FLAG: --cloud-config=""
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545382    2563 flags.go:64] FLAG: --cloud-provider="external"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545385    2563 flags.go:64] FLAG: --cluster-dns="[]"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545389    2563 flags.go:64] FLAG: --cluster-domain=""
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545392    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545395    2563 flags.go:64] FLAG: --config-dir=""
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545399    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545402    2563 flags.go:64] FLAG: --container-log-max-files="5"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545406    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545410    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545413    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545417    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545420    2563 flags.go:64] FLAG: --contention-profiling="false"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545423    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545426    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545429    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545432    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 26 11:24:27.548451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545436    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545439    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545442    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545445    2563 flags.go:64] FLAG: --enable-load-reader="false"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545448    2563 flags.go:64] FLAG: --enable-server="true"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545451    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545481    2563 flags.go:64] FLAG: --event-burst="100"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545485    2563 flags.go:64] FLAG: --event-qps="50"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545488    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545492    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545495    2563 flags.go:64] FLAG: --eviction-hard=""
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545499    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545503    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545506    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545509    2563 flags.go:64] FLAG: --eviction-soft=""
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545512    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545515    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545518    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545521    2563 flags.go:64] FLAG: --experimental-mounter-path=""
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545524    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545527    2563 flags.go:64] FLAG: --fail-swap-on="true"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545530    2563 flags.go:64] FLAG: --feature-gates=""
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545535    2563 flags.go:64] FLAG: --file-check-frequency="20s"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545538    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545541    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 26 11:24:27.549052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545544    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545548    2563 flags.go:64] FLAG: --healthz-port="10248"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545551    2563 flags.go:64] FLAG: --help="false"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545554    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-135-71.ec2.internal"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545557    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545560    2563 flags.go:64] FLAG: --http-check-frequency="20s"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545563    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545567    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545570    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545573    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545576    2563 flags.go:64] FLAG: --image-service-endpoint=""
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545579    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545581    2563 flags.go:64] FLAG: --kube-api-burst="100"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545585    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545588    2563 flags.go:64] FLAG: --kube-api-qps="50"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545591    2563 flags.go:64] FLAG: --kube-reserved=""
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545594    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545597    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545600    2563 flags.go:64] FLAG: --kubelet-cgroups=""
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545603    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545606    2563 flags.go:64] FLAG: --lock-file=""
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545609    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545612    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545618    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 26 11:24:27.549674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545624    2563 flags.go:64] FLAG: --log-json-split-stream="false"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545627    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545630    2563 flags.go:64] FLAG: --log-text-split-stream="false"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545633    2563 flags.go:64] FLAG: --logging-format="text"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545636    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545639    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545642    2563 flags.go:64] FLAG: --manifest-url=""
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545645    2563 flags.go:64] FLAG: --manifest-url-header=""
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545650    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545654    2563 flags.go:64] FLAG: --max-open-files="1000000"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545658    2563 flags.go:64] FLAG: --max-pods="110"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545661    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545664    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545667    2563 flags.go:64] FLAG: --memory-manager-policy="None"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545670    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545673    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545676    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545679    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545687    2563 flags.go:64] FLAG: --node-status-max-images="50"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545690    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545693    2563 flags.go:64] FLAG: --oom-score-adj="-999"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545698    2563 flags.go:64] FLAG: --pod-cidr=""
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545701    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 26 11:24:27.550259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545706    2563 flags.go:64] FLAG: --pod-manifest-path=""
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545709    2563 flags.go:64] FLAG: --pod-max-pids="-1"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545712    2563 flags.go:64] FLAG: --pods-per-core="0"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545715    2563 flags.go:64] FLAG: --port="10250"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545718    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545721    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0422298db989c9875"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545724    2563 flags.go:64] FLAG: --qos-reserved=""
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545727    2563 flags.go:64] FLAG: --read-only-port="10255"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545730    2563 flags.go:64] FLAG: --register-node="true"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545734    2563 flags.go:64] FLAG: --register-schedulable="true"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545737    2563 flags.go:64] FLAG: --register-with-taints=""
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545741    2563 flags.go:64] FLAG: --registry-burst="10"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545744    2563 flags.go:64] FLAG: --registry-qps="5"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545747    2563 flags.go:64] FLAG: --reserved-cpus=""
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545749    2563 flags.go:64] FLAG: --reserved-memory=""
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545753    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545756    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545760    2563 flags.go:64] FLAG: --rotate-certificates="false"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545762    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545766    2563 flags.go:64] FLAG: --runonce="false"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545769    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545773    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545775    2563 flags.go:64] FLAG: --seccomp-default="false"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545778    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545781    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545785    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 26 11:24:27.550817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545789    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545792    2563 flags.go:64] FLAG: --storage-driver-password="root"
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545795    2563 flags.go:64] FLAG: --storage-driver-secure="false"
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545797    2563 flags.go:64] FLAG: --storage-driver-table="stats"
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545813    2563 flags.go:64] FLAG: --storage-driver-user="root"
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545817    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545820    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545823    2563 flags.go:64] FLAG: --system-cgroups=""
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545827    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545832    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545835    2563 flags.go:64] FLAG: --tls-cert-file=""
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545838    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545843    2563 flags.go:64] FLAG: --tls-min-version=""
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545846    2563 flags.go:64] FLAG: --tls-private-key-file=""
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545849    2563 flags.go:64] FLAG: --topology-manager-policy="none"
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545853    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545857    2563 flags.go:64] FLAG: --topology-manager-scope="container"
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545860    2563 flags.go:64] FLAG: --v="2"
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545865    2563 flags.go:64] FLAG: --version="false"
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545869    2563 flags.go:64] FLAG: --vmodule=""
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545873    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.545876    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.545994    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.545998    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:27.551488 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546001    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546004    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546007    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546009    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546013    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546015    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546018    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546021    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546023    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546026    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546028    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546031    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546034    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546037    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546040    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546042    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546045    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546047    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546050    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546053    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:27.552104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546055    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546074    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546078    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546082    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546085    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546087    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546090    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546093    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546095    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546098    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546100    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546103    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546105    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546108    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546110    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546114    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546116    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546119    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546121    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:27.552637 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546125    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546128    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546131    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546133    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546136    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546143    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546146    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546148    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546151    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546154    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546156    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546159    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546161    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546164    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546166    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546169    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546172    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546175    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546177    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546180    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:27.553118 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546183    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546185    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546188    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546191    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546193    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546196    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546198    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546201    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546203    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546206    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546209    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546212    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546214    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546216    2563 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546219    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546221    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546224    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546228    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546230    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:27.553629 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546234    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:27.554103 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546238    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:27.554103 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546241    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:27.554103 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546244    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:27.554103 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546247    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:27.554103 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.546250    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:27.554103 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.546255    2563 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:24:27.554103 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.552522    2563 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 26 11:24:27.554103 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.552538    2563 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 26 11:24:27.554103 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552583    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:27.554103 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552588    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:27.554103 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552591    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:27.554103 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552594    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:27.554103 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552597    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:27.554103 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552600    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:27.554103 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552603    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552606    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552609    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552612    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552615    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552617    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552620    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552623    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552625    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552628    2563 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552631    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552634    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552636    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552639    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552642    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552646    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552648    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552651    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552654    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:27.554512 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552656    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552659    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552661    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552664    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552668    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552671    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552675    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552678    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552681    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552683    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552686    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552688    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552691    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552693    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552696    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552698    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552701    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552704    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552706    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552709    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:27.554982 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552711    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552713    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552716    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552719    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552721    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552724    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552726    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552728    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552731    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552733    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552736    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552739    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552741    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552744    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552747    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552750    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552752    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552755    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552757    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552760    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:27.555521 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552763    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552765    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552768    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552770    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552773    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552776    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552778    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552780    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552783    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552786    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552789    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552791    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552793    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552796    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552798    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552803    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552807    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552810    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552813    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:27.556012 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552816    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:27.556517 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552819    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:27.556517 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.552825    2563 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:24:27.556517 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552920    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:27.556517 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552924    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:27.556517 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552927    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:27.556517 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552930    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:27.556517 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552933    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:27.556517 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552935    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:27.556517 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552938    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:27.556517 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552941    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:27.556517 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552944    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:27.556517 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552947    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:27.556517 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552950    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:27.556517 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552953    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:27.556517 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552955    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:27.556517 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552958    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552961    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552963    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552966    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552968    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552971    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552974    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552976    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552978    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552981    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552983    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552986    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552989    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552991    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552994    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.552996    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553000    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553004    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553007    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:27.556930 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553010    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553013    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553015    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553018    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553020    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553023    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553025    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553028    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553031    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553033    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553036    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553039    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553041    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553044    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553046    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553049    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553051    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553054    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553056    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553059    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:27.557423 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553061    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553064    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553067    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553069    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553072    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553074    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553076    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553079    2563 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553081    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553084    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553087    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553089    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553092    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553094    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553097    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553099    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553102    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553105    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553107    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553109    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:27.557919 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553112    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:27.558475 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553115    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:27.558475 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553117    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:27.558475 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553120    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:27.558475 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553124    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:27.558475 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553127    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:27.558475 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553130    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:27.558475 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553132    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:27.558475 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553135    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:27.558475 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553137    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:27.558475 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553141    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:27.558475 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553144    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:27.558475 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553147    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:27.558475 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:27.553150    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:27.558475 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.553155    2563 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:24:27.558475 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.553895    2563 server.go:962] "Client rotation is on, will bootstrap in background"
May 26 11:24:27.558848 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.555989    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 26 11:24:27.558848 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.556992    2563 server.go:1019] "Starting client certificate rotation"
May 26 11:24:27.558848 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.557091    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 11:24:27.558848 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.557130    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 11:24:27.582826 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.582805    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 11:24:27.591551 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.591524    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 11:24:27.610299 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.610267    2563 log.go:25] "Validated CRI v1 runtime API"
May 26 11:24:27.615531 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.615510    2563 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 11:24:27.615890 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.615875    2563 log.go:25] "Validated CRI v1 image API"
May 26 11:24:27.617059 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.617038    2563 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 26 11:24:27.619386 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.619366    2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a685d673-58e0-4fdb-aaf4-e0f592d5681a:/dev/nvme0n1p4 f77baff1-2ee7-4a18-a63a-7ef4218b8882:/dev/nvme0n1p3]
May 26 11:24:27.619449 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.619385    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 26 11:24:27.623670 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.623563    2563 manager.go:217] Machine: {Timestamp:2026-05-26 11:24:27.623040843 +0000 UTC m=+0.428879298 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3112019 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2487e62afe1234276a8ecd91eef230 SystemUUID:ec2487e6-2afe-1234-276a-8ecd91eef230 BootID:8a29ff84-e8a1-4407-b3e2-49939e313cc3 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:94:ca:b4:fa:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:94:ca:b4:fa:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:56:94:67:1a:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 26 11:24:27.624167 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.624156    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 26 11:24:27.624263 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.624250    2563 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 26 11:24:27.626777 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.626750    2563 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 26 11:24:27.626945 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.626778    2563 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-71.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 26 11:24:27.627024 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.626956    2563 topology_manager.go:138] "Creating topology manager with none policy"
May 26 11:24:27.627024 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.626968    2563 container_manager_linux.go:306] "Creating device plugin manager"
May 26 11:24:27.627024 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.626978    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 11:24:27.627932 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.627917    2563 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 11:24:27.629905 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.629892    2563 state_mem.go:36] "Initialized new in-memory state store"
May 26 11:24:27.630040 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.630029    2563 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 26 11:24:27.632220 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.632208    2563 kubelet.go:491] "Attempting to sync node with API server"
May 26 11:24:27.632292 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.632233    2563 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 26 11:24:27.632292 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.632251    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 26 11:24:27.632292 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.632265    2563 kubelet.go:397] "Adding apiserver pod source"
May 26 11:24:27.632292 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.632291    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 26 11:24:27.633451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.633437    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 11:24:27.633516 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.633461    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 11:24:27.636506 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.636489    2563 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 26 11:24:27.637992 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.637792    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-59556"
May 26 11:24:27.638419 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.638406    2563 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 26 11:24:27.639468 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.639365    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 26 11:24:27.639468 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.639386    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 26 11:24:27.639468 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.639393    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 26 11:24:27.639468 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.639407    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 26 11:24:27.639468 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.639416    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 26 11:24:27.639468 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.639422    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 26 11:24:27.639468 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.639433    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 26 11:24:27.639468 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.639439    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 26 11:24:27.639468 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.639446    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 26 11:24:27.639468 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.639452    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 26 11:24:27.639468 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.639460    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 26 11:24:27.639765 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.639660    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 26 11:24:27.640527 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.640517    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 26 11:24:27.640527 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.640527    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 26 11:24:27.644078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.644055    2563 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 26 11:24:27.644168 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.644147    2563 server.go:1295] "Started kubelet"
May 26 11:24:27.644655 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.644604    2563 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 26 11:24:27.644748 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.644668    2563 server_v1.go:47] "podresources" method="list" useActivePods=true
May 26 11:24:27.645142 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.645054    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-59556"
May 26 11:24:27.645126 ip-10-0-135-71 systemd[1]: Started Kubernetes Kubelet.
May 26 11:24:27.645721 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.645240    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 26 11:24:27.646189 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.646141    2563 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 26 11:24:27.646784 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.646756    2563 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-71.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 26 11:24:27.646847 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:27.646775    2563 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 26 11:24:27.646847 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:27.646809    2563 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 26 11:24:27.650591 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.650573    2563 server.go:317] "Adding debug handlers to kubelet server"
May 26 11:24:27.653376 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:27.652442    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-71.ec2.internal.18b319d20a0f39b2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-71.ec2.internal,UID:ip-10-0-135-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-71.ec2.internal,},FirstTimestamp:2026-05-26 11:24:27.644074418 +0000 UTC m=+0.449912875,LastTimestamp:2026-05-26 11:24:27.644074418 +0000 UTC m=+0.449912875,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-71.ec2.internal,}"
May 26 11:24:27.655536 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.655519    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 26 11:24:27.655536 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.655531    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 26 11:24:27.655689 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.655622    2563 volume_manager.go:295] "The desired_state_of_world populator starts"
May 26 11:24:27.655689 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.655632    2563 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 26 11:24:27.655784 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.655691    2563 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 26 11:24:27.655833 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.655797    2563 reconstruct.go:97] "Volume reconstruction finished"
May 26 11:24:27.655833 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.655811    2563 reconciler.go:26] "Reconciler: start to sync state"
May 26 11:24:27.656383 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:27.656361    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 26 11:24:27.656565 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.656549    2563 factory.go:55] Registering systemd factory
May 26 11:24:27.656636 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.656599    2563 factory.go:223] Registration of the systemd container factory successfully
May 26 11:24:27.656800 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.656785    2563 factory.go:153] Registering CRI-O factory
May 26 11:24:27.656800 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.656801    2563 factory.go:223] Registration of the crio container factory successfully
May 26 11:24:27.656912 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.656840    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 26 11:24:27.656912 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.656859    2563 factory.go:103] Registering Raw factory
May 26 11:24:27.656912 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.656874    2563 manager.go:1196] Started watching for new ooms in manager
May 26 11:24:27.657167 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:27.657144    2563 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 26 11:24:27.657516 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.657501    2563 manager.go:319] Starting recovery of all containers
May 26 11:24:27.658648 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.658631    2563 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:27.661660 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:27.661633    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-71.ec2.internal\" not found" node="ip-10-0-135-71.ec2.internal"
May 26 11:24:27.667988 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.667845    2563 manager.go:324] Recovery completed
May 26 11:24:27.673266 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.673021    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:27.674836 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.674822    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:27.674924 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.674852    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:27.674924 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.674867    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:27.675342 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.675320    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 26 11:24:27.675342 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.675334    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 26 11:24:27.675466 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.675358    2563 state_mem.go:36] "Initialized new in-memory state store"
May 26 11:24:27.677622 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.677608    2563 policy_none.go:49] "None policy: Start"
May 26 11:24:27.677697 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.677626    2563 memory_manager.go:186] "Starting memorymanager" policy="None"
May 26 11:24:27.677697 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.677643    2563 state_mem.go:35] "Initializing new in-memory state store"
May 26 11:24:27.716884 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.716868    2563 manager.go:341] "Starting Device Plugin manager"
May 26 11:24:27.722161 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:27.716902    2563 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 26 11:24:27.722161 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.716914    2563 server.go:85] "Starting device plugin registration server"
May 26 11:24:27.722161 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.717122    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
May 26 11:24:27.722161 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.717133    2563 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 26 11:24:27.722161 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.717227    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 26 11:24:27.722161 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.717343    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 26 11:24:27.722161 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.717353    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 26 11:24:27.722161 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:27.717899    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 26 11:24:27.722161 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:27.717930    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-71.ec2.internal\" not found"
May 26 11:24:27.789716 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.789647    2563 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 26 11:24:27.791001 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.790984    2563 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 26 11:24:27.791093 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.791007    2563 status_manager.go:230] "Starting to sync pod status with apiserver"
May 26 11:24:27.791093 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.791024    2563 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 26 11:24:27.791093 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.791032    2563 kubelet.go:2451] "Starting kubelet main sync loop"
May 26 11:24:27.791093 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:27.791075    2563 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 26 11:24:27.793831 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.793815    2563 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:27.817826 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.817809    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:27.818590 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.818574    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:27.818677 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.818606    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:27.818677 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.818621    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:27.818677 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.818651    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-71.ec2.internal"
May 26 11:24:27.826999 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.826984    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-71.ec2.internal"
May 26 11:24:27.827055 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:27.827006    2563 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-71.ec2.internal\": node \"ip-10-0-135-71.ec2.internal\" not found"
May 26 11:24:27.839843 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:27.839824    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 26 11:24:27.891569 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.891544    2563 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"]
May 26 11:24:27.891650 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.891613    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:27.892545 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.892530    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:27.892607 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.892557    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:27.892607 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.892566    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:27.893798 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.893787    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:27.893925 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.893909    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal"
May 26 11:24:27.893977 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.893944    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:27.894540 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.894524    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:27.894540 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.894527    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:27.894650 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.894546    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:27.894650 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.894560    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:27.894650 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.894566    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:27.894650 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.894582    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:27.895802 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.895789    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 26 11:24:27.895854 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.895820    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:27.896489 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.896468    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:27.896578 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.896495    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:27.896578 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.896513    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:27.914826 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:27.914803    2563 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-71.ec2.internal\" not found" node="ip-10-0-135-71.ec2.internal"
May 26 11:24:27.918756 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:27.918741    2563 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-71.ec2.internal\" not found" node="ip-10-0-135-71.ec2.internal"
May 26 11:24:27.940168 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:27.940143    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 26 11:24:27.957712 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.957692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a50d4f711fbf255cb7babbc8960a87b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-71.ec2.internal\" (UID: \"9a50d4f711fbf255cb7babbc8960a87b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal"
May 26 11:24:27.957772 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.957719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dcd208f0122d6cb3e226ed802a3b1ab4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal\" (UID: \"dcd208f0122d6cb3e226ed802a3b1ab4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 26 11:24:27.957772 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:27.957737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcd208f0122d6cb3e226ed802a3b1ab4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal\" (UID: \"dcd208f0122d6cb3e226ed802a3b1ab4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 26 11:24:28.040403 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.040339    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 26 11:24:28.058704 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.058672    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a50d4f711fbf255cb7babbc8960a87b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-71.ec2.internal\" (UID: \"9a50d4f711fbf255cb7babbc8960a87b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal"
May 26 11:24:28.058704 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.058704    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dcd208f0122d6cb3e226ed802a3b1ab4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal\" (UID: \"dcd208f0122d6cb3e226ed802a3b1ab4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 26 11:24:28.058858 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.058721    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcd208f0122d6cb3e226ed802a3b1ab4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal\" (UID: \"dcd208f0122d6cb3e226ed802a3b1ab4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 26 11:24:28.058858 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.058772    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a50d4f711fbf255cb7babbc8960a87b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-71.ec2.internal\" (UID: \"9a50d4f711fbf255cb7babbc8960a87b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal"
May 26 11:24:28.058858 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.058824    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcd208f0122d6cb3e226ed802a3b1ab4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal\" (UID: \"dcd208f0122d6cb3e226ed802a3b1ab4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 26 11:24:28.058858 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.058850    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dcd208f0122d6cb3e226ed802a3b1ab4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal\" (UID: \"dcd208f0122d6cb3e226ed802a3b1ab4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 26 11:24:28.141149 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.141111    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 26 11:24:28.217561 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.217532    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal"
May 26 11:24:28.222319 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.222276    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 26 11:24:28.242177 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.242143    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 26 11:24:28.342680 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.342602    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 26 11:24:28.443096 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.443070    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 26 11:24:28.543632 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.543602    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-71.ec2.internal\" not found"
May 26 11:24:28.556775 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.556751    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 26 11:24:28.556957 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.556938    2563 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 11:24:28.556995 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.556938    2563 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 11:24:28.579509 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.579484    2563 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:28.632597 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.632565    2563 apiserver.go:52] "Watching apiserver"
May 26 11:24:28.641809 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.641787    2563 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 26 11:24:28.642619 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.642591    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-zk6bl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx","openshift-cluster-node-tuning-operator/tuned-s4hrh","openshift-multus/multus-additional-cni-plugins-7rmwg","openshift-multus/multus-fhpf8","openshift-network-diagnostics/network-check-target-5pwxn","openshift-network-operator/iptables-alerter-gv6w6","openshift-ovn-kubernetes/ovnkube-node-dkfdl","kube-system/global-pull-secret-syncer-wzshh","openshift-image-registry/node-ca-8dl7t","openshift-multus/network-metrics-daemon-vgwsb"]
May 26 11:24:28.646666 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.646644    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zk6bl"
May 26 11:24:28.647931 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.647660    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.647931 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.647733    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.648895 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.648876    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 26 11:24:28.648895 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.648890    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5pwbx\""
May 26 11:24:28.649012 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.648890    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 26 11:24:28.649842 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.649762    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 26 11:24:28.649961 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.649947    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b7x8p\""
May 26 11:24:28.650001 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.649971    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 26 11:24:28.650089 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.650061    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.650243 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.650139    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 26 11:24:28.650243 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.650161    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f28xs\""
May 26 11:24:28.650243 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.650163    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 26 11:24:28.650418 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.650372    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 26 11:24:28.650742 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.650717    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-25 11:19:27 +0000 UTC" deadline="2027-10-22 02:47:58.87884512 +0000 UTC"
May 26 11:24:28.650742 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.650739    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12327h23m30.228108567s"
May 26 11:24:28.651473 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.651457    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.651646 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.651607    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:28.651715 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.651691    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:28.652758 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.652736    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wr99k\""
May 26 11:24:28.652844 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.652798    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gv6w6"
May 26 11:24:28.653013 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.652997    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 26 11:24:28.653187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.653172    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 26 11:24:28.653262 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.653247    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 26 11:24:28.653491 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.653477    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nkhjs\""
May 26 11:24:28.653580 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.653564    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 26 11:24:28.653680 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.653666    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 26 11:24:28.654107 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.654092    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.654833 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.654804    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 26 11:24:28.654928 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.654888    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 26 11:24:28.654928 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.654906    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 26 11:24:28.654928 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.654908    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-czw2q\""
May 26 11:24:28.655262 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.655239    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 26 11:24:28.655365 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.655349    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8dl7t"
May 26 11:24:28.655623 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.655583    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 26 11:24:28.655859 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.655846    2563 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"
May 26 11:24:28.656259 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.656215    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6b4d4\""
May 26 11:24:28.656369 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.656352    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 26 11:24:28.656427 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.656353    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 26 11:24:28.656483 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.656444    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 26 11:24:28.656565 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.656546    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:28.656640 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.656621    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:24:28.657096 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.657078    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 26 11:24:28.657213 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.657115    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 26 11:24:28.657213 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.657124    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 26 11:24:28.657213 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.657179    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 26 11:24:28.657435 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.657420    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 26 11:24:28.658095 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.657928    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 26 11:24:28.658095 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.657941    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xg9c2\""
May 26 11:24:28.658095 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.657968    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:28.658095 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.658038    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:28.662494 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f5lg\" (UniqueName: \"kubernetes.io/projected/1061f872-9b39-430c-9fe2-e1308e621e42-kube-api-access-6f5lg\") pod \"node-ca-8dl7t\" (UID: \"1061f872-9b39-430c-9fe2-e1308e621e42\") " pod="openshift-image-registry/node-ca-8dl7t"
May 26 11:24:28.662574 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c2dae63-45e0-481d-b9de-7eea78edbc6b-iptables-alerter-script\") pod \"iptables-alerter-gv6w6\" (UID: \"1c2dae63-45e0-481d-b9de-7eea78edbc6b\") " pod="openshift-network-operator/iptables-alerter-gv6w6"
May 26 11:24:28.662574 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-system-cni-dir\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.662574 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-multus-cni-dir\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.662673 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-multus-daemon-config\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.662673 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662602    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-run-openvswitch\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.662673 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-ovn-node-metrics-cert\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.662673 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gwc9\" (UniqueName: \"kubernetes.io/projected/daf3e280-5478-4519-83a2-5e5a7af26aa4-kube-api-access-6gwc9\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.662860 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662706    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-etc-kubernetes\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.662860 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-run-netns\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.662860 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1061f872-9b39-430c-9fe2-e1308e621e42-host\") pod \"node-ca-8dl7t\" (UID: \"1061f872-9b39-430c-9fe2-e1308e621e42\") " pod="openshift-image-registry/node-ca-8dl7t"
May 26 11:24:28.662860 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grs2l\" (UniqueName: \"kubernetes.io/projected/1c2dae63-45e0-481d-b9de-7eea78edbc6b-kube-api-access-grs2l\") pod \"iptables-alerter-gv6w6\" (UID: \"1c2dae63-45e0-481d-b9de-7eea78edbc6b\") " pod="openshift-network-operator/iptables-alerter-gv6w6"
May 26 11:24:28.663053 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662868    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-sys\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.663053 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a6b09f3-e909-4c40-a22c-a6dd5beab731-tmp\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.663053 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-cnibin\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.663053 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662918    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-run-netns\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.663053 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-var-lib-kubelet\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.663053 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662961    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd8ad6b4-eb8d-41fe-8247-de669518e6d4-agent-certs\") pod \"konnectivity-agent-zk6bl\" (UID: \"bd8ad6b4-eb8d-41fe-8247-de669518e6d4\") " pod="kube-system/konnectivity-agent-zk6bl"
May 26 11:24:28.663053 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-run\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.663053 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.662994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/daf3e280-5478-4519-83a2-5e5a7af26aa4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.663053 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-node-log\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.663053 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.663053 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wk2l\" (UniqueName: \"kubernetes.io/projected/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-kube-api-access-7wk2l\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.663555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frw6r\" (UniqueName: \"kubernetes.io/projected/a9201d70-b657-4adf-9d01-5363b9fb7ded-kube-api-access-frw6r\") pod \"network-metrics-daemon-vgwsb\" (UID: \"a9201d70-b657-4adf-9d01-5363b9fb7ded\") " pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:28.663555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-kubernetes\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.663555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/daf3e280-5478-4519-83a2-5e5a7af26aa4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.663555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52sp9\" (UniqueName: \"kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9\") pod \"network-check-target-5pwxn\" (UID: \"44da6752-3b21-42e6-942c-ab7fc44713dc\") " pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:28.663555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-etc-openvswitch\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.663555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1061f872-9b39-430c-9fe2-e1308e621e42-serviceca\") pod \"node-ca-8dl7t\" (UID: \"1061f872-9b39-430c-9fe2-e1308e621e42\") " pod="openshift-image-registry/node-ca-8dl7t"
May 26 11:24:28.663555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-tuned\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.663555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-cni-binary-copy\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.663555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663314    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-multus-conf-dir\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.663555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-var-lib-openvswitch\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.663555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-run-ovn\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.663555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k77hv\" (UniqueName: \"kubernetes.io/projected/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-kube-api-access-k77hv\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.663555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-multus-socket-dir-parent\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.663555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-var-lib-cni-multus\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.663555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-slash\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.663555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd8ad6b4-eb8d-41fe-8247-de669518e6d4-konnectivity-ca\") pod \"konnectivity-agent-zk6bl\" (UID: \"bd8ad6b4-eb8d-41fe-8247-de669518e6d4\") " pod="kube-system/konnectivity-agent-zk6bl"
May 26 11:24:28.664187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-socket-dir\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.664187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-sys-fs\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.664187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663574    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c2dae63-45e0-481d-b9de-7eea78edbc6b-host-slash\") pod \"iptables-alerter-gv6w6\" (UID: \"1c2dae63-45e0-481d-b9de-7eea78edbc6b\") " pod="openshift-network-operator/iptables-alerter-gv6w6"
May 26 11:24:28.664187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-sysconfig\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.664187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/daf3e280-5478-4519-83a2-5e5a7af26aa4-cnibin\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.664187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-run-ovn-kubernetes\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.664187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-cni-bin\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.664187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-modprobe-d\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.664187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-sysctl-conf\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.664187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td872\" (UniqueName: \"kubernetes.io/projected/7a6b09f3-e909-4c40-a22c-a6dd5beab731-kube-api-access-td872\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.664187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/daf3e280-5478-4519-83a2-5e5a7af26aa4-system-cni-dir\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.664187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-os-release\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.664187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4sz4\" (UniqueName: \"kubernetes.io/projected/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-kube-api-access-p4sz4\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.664187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-run-systemd\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.664187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-ovnkube-config\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.664187 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.663984    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-var-lib-kubelet\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.664886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664014    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-host\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.664886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/daf3e280-5478-4519-83a2-5e5a7af26aa4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.664886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-registration-dir\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.664886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-device-dir\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.664886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-sysctl-d\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.664886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-run-k8s-cni-cncf-io\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.664886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-cni-netd\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.664886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-lib-modules\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.664886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664214    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-hostroot\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.664886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-kubelet\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.664886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-log-socket\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.664886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs\") pod \"network-metrics-daemon-vgwsb\" (UID: \"a9201d70-b657-4adf-9d01-5363b9fb7ded\") " pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:28.664886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-systemd\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.664886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664344    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/daf3e280-5478-4519-83a2-5e5a7af26aa4-os-release\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.664886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-var-lib-cni-bin\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.664886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-run-multus-certs\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.665478 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-env-overrides\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.665478 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-ovnkube-script-lib\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.665478 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/daf3e280-5478-4519-83a2-5e5a7af26aa4-cni-binary-copy\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.665478 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-systemd-units\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.665478 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.665478 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.664634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.669487 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.669468    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal"]
May 26 11:24:28.670669 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.670653    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 11:24:28.670748 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.670723    2563 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal"
May 26 11:24:28.676647 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.676631    2563 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 11:24:28.679227 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.679210    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 11:24:28.679734 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.679715    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal"]
May 26 11:24:28.699397 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.699376    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qtw52"
May 26 11:24:28.707861 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.707841    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qtw52"
May 26 11:24:28.739429 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:28.739388    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd208f0122d6cb3e226ed802a3b1ab4.slice/crio-fdb4ac93d9f342bf8778b7c9fcb9d7557fca516586290df19eb570bcfcf5c0fe WatchSource:0}: Error finding container fdb4ac93d9f342bf8778b7c9fcb9d7557fca516586290df19eb570bcfcf5c0fe: Status 404 returned error can't find the container with id fdb4ac93d9f342bf8778b7c9fcb9d7557fca516586290df19eb570bcfcf5c0fe
May 26 11:24:28.739713 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:28.739689    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a50d4f711fbf255cb7babbc8960a87b.slice/crio-693e69852e7977388be708f54018008f4e97575e5add742371aa7ddb26a1d96b WatchSource:0}: Error finding container 693e69852e7977388be708f54018008f4e97575e5add742371aa7ddb26a1d96b: Status 404 returned error can't find the container with id 693e69852e7977388be708f54018008f4e97575e5add742371aa7ddb26a1d96b
May 26 11:24:28.743420 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.743406    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:24:28.757067 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.757024    2563 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 26 11:24:28.765421 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765401    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd8ad6b4-eb8d-41fe-8247-de669518e6d4-konnectivity-ca\") pod \"konnectivity-agent-zk6bl\" (UID: \"bd8ad6b4-eb8d-41fe-8247-de669518e6d4\") " pod="kube-system/konnectivity-agent-zk6bl"
May 26 11:24:28.765513 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765427    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-socket-dir\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.765513 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765446    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-sys-fs\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.765513 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765470    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c2dae63-45e0-481d-b9de-7eea78edbc6b-host-slash\") pod \"iptables-alerter-gv6w6\" (UID: \"1c2dae63-45e0-481d-b9de-7eea78edbc6b\") " pod="openshift-network-operator/iptables-alerter-gv6w6"
May 26 11:24:28.765513 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765493    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-sysconfig\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.765719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765516    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/daf3e280-5478-4519-83a2-5e5a7af26aa4-cnibin\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.765719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765524    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-sys-fs\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.765719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765539    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-run-ovn-kubernetes\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.765719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765572    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-cni-bin\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.765719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765575    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c2dae63-45e0-481d-b9de-7eea78edbc6b-host-slash\") pod \"iptables-alerter-gv6w6\" (UID: \"1c2dae63-45e0-481d-b9de-7eea78edbc6b\") " pod="openshift-network-operator/iptables-alerter-gv6w6"
May 26 11:24:28.765719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765578    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-socket-dir\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.765719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765595    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-modprobe-d\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.765719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765615    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-run-ovn-kubernetes\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.765719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765616    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-sysconfig\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.765719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765648    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-sysctl-conf\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.765719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765669    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-td872\" (UniqueName: \"kubernetes.io/projected/7a6b09f3-e909-4c40-a22c-a6dd5beab731-kube-api-access-td872\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.765719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765678    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/daf3e280-5478-4519-83a2-5e5a7af26aa4-cnibin\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.765719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765681    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-modprobe-d\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.765719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765690    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/daf3e280-5478-4519-83a2-5e5a7af26aa4-system-cni-dir\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.765719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765716    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/daf3e280-5478-4519-83a2-5e5a7af26aa4-system-cni-dir\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765729    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-os-release\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765746    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-cni-bin\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765755    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p4sz4\" (UniqueName: \"kubernetes.io/projected/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-kube-api-access-p4sz4\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765773    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-sysctl-conf\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765799    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-os-release\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765806    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-run-systemd\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765780    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-run-systemd\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765838    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-ovnkube-config\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765862    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-var-lib-kubelet\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765884    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-host\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765908    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/daf3e280-5478-4519-83a2-5e5a7af26aa4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765923    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd8ad6b4-eb8d-41fe-8247-de669518e6d4-konnectivity-ca\") pod \"konnectivity-agent-zk6bl\" (UID: \"bd8ad6b4-eb8d-41fe-8247-de669518e6d4\") " pod="kube-system/konnectivity-agent-zk6bl"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765933    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-registration-dir\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765962    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-device-dir\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f39775e-7a53-44fb-8e18-70fd6a5764c5-dbus\") pod \"global-pull-secret-syncer-wzshh\" (UID: \"7f39775e-7a53-44fb-8e18-70fd6a5764c5\") " pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.765987    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-var-lib-kubelet\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.766462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret\") pod \"global-pull-secret-syncer-wzshh\" (UID: \"7f39775e-7a53-44fb-8e18-70fd6a5764c5\") " pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766095    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/daf3e280-5478-4519-83a2-5e5a7af26aa4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766046    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-registration-dir\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766074    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-device-dir\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766123    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-sysctl-d\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766041    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-host\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766172    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-run-k8s-cni-cncf-io\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766203    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-cni-netd\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766191    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-run-k8s-cni-cncf-io\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766245    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-sysctl-d\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766229    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-lib-modules\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766265    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-cni-netd\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766319    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-hostroot\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766329    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-lib-modules\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766345    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-kubelet\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766359    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-hostroot\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766369    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-log-socket\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766391    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-kubelet\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.767331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766395    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs\") pod \"network-metrics-daemon-vgwsb\" (UID: \"a9201d70-b657-4adf-9d01-5363b9fb7ded\") " pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766429    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-systemd\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766451    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/daf3e280-5478-4519-83a2-5e5a7af26aa4-os-release\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.766471    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766474    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-var-lib-cni-bin\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766481    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-ovnkube-config\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766500    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-run-multus-certs\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.766529    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs podName:a9201d70-b657-4adf-9d01-5363b9fb7ded nodeName:}" failed. No retries permitted until 2026-05-26 11:24:29.266506934 +0000 UTC m=+2.072345391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs") pod "network-metrics-daemon-vgwsb" (UID: "a9201d70-b657-4adf-9d01-5363b9fb7ded") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766536    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-run-multus-certs\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766565    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-var-lib-cni-bin\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766586    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/daf3e280-5478-4519-83a2-5e5a7af26aa4-os-release\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766603    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-systemd\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766525    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-log-socket\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766634    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-env-overrides\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766657    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-ovnkube-script-lib\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766694    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/daf3e280-5478-4519-83a2-5e5a7af26aa4-cni-binary-copy\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766711    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-systemd-units\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.768078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766726    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766742    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766763    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5lg\" (UniqueName: \"kubernetes.io/projected/1061f872-9b39-430c-9fe2-e1308e621e42-kube-api-access-6f5lg\") pod \"node-ca-8dl7t\" (UID: \"1061f872-9b39-430c-9fe2-e1308e621e42\") " pod="openshift-image-registry/node-ca-8dl7t"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766791    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c2dae63-45e0-481d-b9de-7eea78edbc6b-iptables-alerter-script\") pod \"iptables-alerter-gv6w6\" (UID: \"1c2dae63-45e0-481d-b9de-7eea78edbc6b\") " pod="openshift-network-operator/iptables-alerter-gv6w6"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766819    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-system-cni-dir\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766845    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-multus-cni-dir\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766850    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766868    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-multus-daemon-config\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766890    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-run-openvswitch\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.766944    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-run-openvswitch\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767053    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-env-overrides\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767106    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-ovn-node-metrics-cert\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767132    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-ovnkube-script-lib\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767141    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f39775e-7a53-44fb-8e18-70fd6a5764c5-kubelet-config\") pod \"global-pull-secret-syncer-wzshh\" (UID: \"7f39775e-7a53-44fb-8e18-70fd6a5764c5\") " pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767164    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767174    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6gwc9\" (UniqueName: \"kubernetes.io/projected/daf3e280-5478-4519-83a2-5e5a7af26aa4-kube-api-access-6gwc9\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767213    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-systemd-units\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.768797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767380    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/daf3e280-5478-4519-83a2-5e5a7af26aa4-cni-binary-copy\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767423    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-multus-cni-dir\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767420    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c2dae63-45e0-481d-b9de-7eea78edbc6b-iptables-alerter-script\") pod \"iptables-alerter-gv6w6\" (UID: \"1c2dae63-45e0-481d-b9de-7eea78edbc6b\") " pod="openshift-network-operator/iptables-alerter-gv6w6"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767429    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-etc-kubernetes\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767468    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-run-netns\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767469    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-etc-kubernetes\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767469    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-system-cni-dir\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767495    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1061f872-9b39-430c-9fe2-e1308e621e42-host\") pod \"node-ca-8dl7t\" (UID: \"1061f872-9b39-430c-9fe2-e1308e621e42\") " pod="openshift-image-registry/node-ca-8dl7t"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767505    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-run-netns\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767518    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767555    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-grs2l\" (UniqueName: \"kubernetes.io/projected/1c2dae63-45e0-481d-b9de-7eea78edbc6b-kube-api-access-grs2l\") pod \"iptables-alerter-gv6w6\" (UID: \"1c2dae63-45e0-481d-b9de-7eea78edbc6b\") " pod="openshift-network-operator/iptables-alerter-gv6w6"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767574    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-sys\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767599    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a6b09f3-e909-4c40-a22c-a6dd5beab731-tmp\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767617    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1061f872-9b39-430c-9fe2-e1308e621e42-host\") pod \"node-ca-8dl7t\" (UID: \"1061f872-9b39-430c-9fe2-e1308e621e42\") " pod="openshift-image-registry/node-ca-8dl7t"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767623    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-cnibin\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767648    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-run-netns\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767673    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-var-lib-kubelet\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767719    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-multus-daemon-config\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767730    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-run-netns\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.769570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767748    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-cnibin\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767770    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd8ad6b4-eb8d-41fe-8247-de669518e6d4-agent-certs\") pod \"konnectivity-agent-zk6bl\" (UID: \"bd8ad6b4-eb8d-41fe-8247-de669518e6d4\") " pod="kube-system/konnectivity-agent-zk6bl"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767794    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-run\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767891    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/daf3e280-5478-4519-83a2-5e5a7af26aa4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767820    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-sys\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767851    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-run\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767806    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-var-lib-kubelet\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767933    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-node-log\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.767977    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-node-log\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768005    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768032    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7wk2l\" (UniqueName: \"kubernetes.io/projected/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-kube-api-access-7wk2l\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768057    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-frw6r\" (UniqueName: \"kubernetes.io/projected/a9201d70-b657-4adf-9d01-5363b9fb7ded-kube-api-access-frw6r\") pod \"network-metrics-daemon-vgwsb\" (UID: \"a9201d70-b657-4adf-9d01-5363b9fb7ded\") " pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768080    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-kubernetes\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768110    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/daf3e280-5478-4519-83a2-5e5a7af26aa4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768136    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-52sp9\" (UniqueName: \"kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9\") pod \"network-check-target-5pwxn\" (UID: \"44da6752-3b21-42e6-942c-ab7fc44713dc\") " pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768158    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-etc-openvswitch\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768181    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1061f872-9b39-430c-9fe2-e1308e621e42-serviceca\") pod \"node-ca-8dl7t\" (UID: \"1061f872-9b39-430c-9fe2-e1308e621e42\") " pod="openshift-image-registry/node-ca-8dl7t"
May 26 11:24:28.770452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768205    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-tuned\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768226    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-cni-binary-copy\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768250    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-multus-conf-dir\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768313    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-multus-conf-dir\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768325    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768375    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-kubernetes\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768405    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/daf3e280-5478-4519-83a2-5e5a7af26aa4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768524    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-var-lib-openvswitch\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768553    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-run-ovn\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768580    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k77hv\" (UniqueName: \"kubernetes.io/projected/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-kube-api-access-k77hv\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768605    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-multus-socket-dir-parent\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768631    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-var-lib-cni-multus\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768658    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-slash\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768724    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-host-slash\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768765    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-var-lib-openvswitch\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768805    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-run-ovn\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768817    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/daf3e280-5478-4519-83a2-5e5a7af26aa4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.768891    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1061f872-9b39-430c-9fe2-e1308e621e42-serviceca\") pod \"node-ca-8dl7t\" (UID: \"1061f872-9b39-430c-9fe2-e1308e621e42\") " pod="openshift-image-registry/node-ca-8dl7t"
May 26 11:24:28.771052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.769025    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-multus-socket-dir-parent\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.771557 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.769067    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-etc-openvswitch\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.771557 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.769109    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-host-var-lib-cni-multus\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.771557 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.769481    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-cni-binary-copy\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.771557 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.770812    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a6b09f3-e909-4c40-a22c-a6dd5beab731-tmp\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.771557 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.770876    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd8ad6b4-eb8d-41fe-8247-de669518e6d4-agent-certs\") pod \"konnectivity-agent-zk6bl\" (UID: \"bd8ad6b4-eb8d-41fe-8247-de669518e6d4\") " pod="kube-system/konnectivity-agent-zk6bl"
May 26 11:24:28.771557 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.770885    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-ovn-node-metrics-cert\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.771557 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.771091    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a6b09f3-e909-4c40-a22c-a6dd5beab731-etc-tuned\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.775774 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.775755    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:28.775833 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.775779    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:28.775833 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.775791    2563 projected.go:194] Error preparing data for projected volume kube-api-access-52sp9 for pod openshift-network-diagnostics/network-check-target-5pwxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:28.775906 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.775848    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9 podName:44da6752-3b21-42e6-942c-ab7fc44713dc nodeName:}" failed. No retries permitted until 2026-05-26 11:24:29.275831475 +0000 UTC m=+2.081669930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-52sp9" (UniqueName: "kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9") pod "network-check-target-5pwxn" (UID: "44da6752-3b21-42e6-942c-ab7fc44713dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:28.775962 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.775937    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td872\" (UniqueName: \"kubernetes.io/projected/7a6b09f3-e909-4c40-a22c-a6dd5beab731-kube-api-access-td872\") pod \"tuned-s4hrh\" (UID: \"7a6b09f3-e909-4c40-a22c-a6dd5beab731\") " pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:28.776405 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.776386    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4sz4\" (UniqueName: \"kubernetes.io/projected/3b46a615-dbda-42ba-ba5b-0ac8bb3076c8-kube-api-access-p4sz4\") pod \"multus-fhpf8\" (UID: \"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8\") " pod="openshift-multus/multus-fhpf8"
May 26 11:24:28.778094 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.778062    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grs2l\" (UniqueName: \"kubernetes.io/projected/1c2dae63-45e0-481d-b9de-7eea78edbc6b-kube-api-access-grs2l\") pod \"iptables-alerter-gv6w6\" (UID: \"1c2dae63-45e0-481d-b9de-7eea78edbc6b\") " pod="openshift-network-operator/iptables-alerter-gv6w6"
May 26 11:24:28.778350 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.778316    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5lg\" (UniqueName: \"kubernetes.io/projected/1061f872-9b39-430c-9fe2-e1308e621e42-kube-api-access-6f5lg\") pod \"node-ca-8dl7t\" (UID: \"1061f872-9b39-430c-9fe2-e1308e621e42\") " pod="openshift-image-registry/node-ca-8dl7t"
May 26 11:24:28.778829 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.778803    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wk2l\" (UniqueName: \"kubernetes.io/projected/31245d18-ee73-4dd5-9f1e-e0fe150f05d9-kube-api-access-7wk2l\") pod \"aws-ebs-csi-driver-node-5g7kx\" (UID: \"31245d18-ee73-4dd5-9f1e-e0fe150f05d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:28.779130 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.779112    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frw6r\" (UniqueName: \"kubernetes.io/projected/a9201d70-b657-4adf-9d01-5363b9fb7ded-kube-api-access-frw6r\") pod \"network-metrics-daemon-vgwsb\" (UID: \"a9201d70-b657-4adf-9d01-5363b9fb7ded\") " pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:28.779212 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.779152    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k77hv\" (UniqueName: \"kubernetes.io/projected/a1d1b0b3-d529-499b-8126-e90cbc5a8ac4-kube-api-access-k77hv\") pod \"ovnkube-node-dkfdl\" (UID: \"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:28.779447 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.779430    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gwc9\" (UniqueName: \"kubernetes.io/projected/daf3e280-5478-4519-83a2-5e5a7af26aa4-kube-api-access-6gwc9\") pod \"multus-additional-cni-plugins-7rmwg\" (UID: \"daf3e280-5478-4519-83a2-5e5a7af26aa4\") " pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:28.793844 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.793807    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal" event={"ID":"9a50d4f711fbf255cb7babbc8960a87b","Type":"ContainerStarted","Data":"693e69852e7977388be708f54018008f4e97575e5add742371aa7ddb26a1d96b"}
May 26 11:24:28.794753 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.794733    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal" event={"ID":"dcd208f0122d6cb3e226ed802a3b1ab4","Type":"ContainerStarted","Data":"fdb4ac93d9f342bf8778b7c9fcb9d7557fca516586290df19eb570bcfcf5c0fe"}
May 26 11:24:28.869641 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.869561    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f39775e-7a53-44fb-8e18-70fd6a5764c5-dbus\") pod \"global-pull-secret-syncer-wzshh\" (UID: \"7f39775e-7a53-44fb-8e18-70fd6a5764c5\") " pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:28.869641 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.869596    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret\") pod \"global-pull-secret-syncer-wzshh\" (UID: \"7f39775e-7a53-44fb-8e18-70fd6a5764c5\") " pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:28.869827 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.869703    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:24:28.869827 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.869758    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f39775e-7a53-44fb-8e18-70fd6a5764c5-kubelet-config\") pod \"global-pull-secret-syncer-wzshh\" (UID: \"7f39775e-7a53-44fb-8e18-70fd6a5764c5\") " pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:28.869827 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.869760    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f39775e-7a53-44fb-8e18-70fd6a5764c5-dbus\") pod \"global-pull-secret-syncer-wzshh\" (UID: \"7f39775e-7a53-44fb-8e18-70fd6a5764c5\") " pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:28.869827 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:28.869768    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret podName:7f39775e-7a53-44fb-8e18-70fd6a5764c5 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:29.369749597 +0000 UTC m=+2.175588059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret") pod "global-pull-secret-syncer-wzshh" (UID: "7f39775e-7a53-44fb-8e18-70fd6a5764c5") : object "kube-system"/"original-pull-secret" not registered
May 26 11:24:28.870053 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.869710    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f39775e-7a53-44fb-8e18-70fd6a5764c5-kubelet-config\") pod \"global-pull-secret-syncer-wzshh\" (UID: \"7f39775e-7a53-44fb-8e18-70fd6a5764c5\") " pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:28.979201 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.979164    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zk6bl"
May 26 11:24:28.998611 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:28.998577    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx"
May 26 11:24:29.004506 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.004481    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s4hrh"
May 26 11:24:29.004598 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:29.004574    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31245d18_ee73_4dd5_9f1e_e0fe150f05d9.slice/crio-936bd463545257dcdf87300c37006b2341f5f7c44902e227b31e380b9e692367 WatchSource:0}: Error finding container 936bd463545257dcdf87300c37006b2341f5f7c44902e227b31e380b9e692367: Status 404 returned error can't find the container with id 936bd463545257dcdf87300c37006b2341f5f7c44902e227b31e380b9e692367
May 26 11:24:29.010136 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:29.010115    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6b09f3_e909_4c40_a22c_a6dd5beab731.slice/crio-7e46c237ab6b86d923111f1801a36133ff0a4a30a422e6cdff1254f2ac1f1529 WatchSource:0}: Error finding container 7e46c237ab6b86d923111f1801a36133ff0a4a30a422e6cdff1254f2ac1f1529: Status 404 returned error can't find the container with id 7e46c237ab6b86d923111f1801a36133ff0a4a30a422e6cdff1254f2ac1f1529
May 26 11:24:29.046093 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.046060    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fhpf8"
May 26 11:24:29.052835 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.052804    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7rmwg"
May 26 11:24:29.054391 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:29.054350    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b46a615_dbda_42ba_ba5b_0ac8bb3076c8.slice/crio-baa9edf1c2ee51a7e598f9481ad564322f38f4ec200ff8dbd471360d5f1fb65a WatchSource:0}: Error finding container baa9edf1c2ee51a7e598f9481ad564322f38f4ec200ff8dbd471360d5f1fb65a: Status 404 returned error can't find the container with id baa9edf1c2ee51a7e598f9481ad564322f38f4ec200ff8dbd471360d5f1fb65a
May 26 11:24:29.058933 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.058905    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gv6w6"
May 26 11:24:29.060308 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:29.060274    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf3e280_5478_4519_83a2_5e5a7af26aa4.slice/crio-9db80ec38ca7dc11577cfcc2f24bb50a75a6d8c6f75ba7f7b3188051dbd7a712 WatchSource:0}: Error finding container 9db80ec38ca7dc11577cfcc2f24bb50a75a6d8c6f75ba7f7b3188051dbd7a712: Status 404 returned error can't find the container with id 9db80ec38ca7dc11577cfcc2f24bb50a75a6d8c6f75ba7f7b3188051dbd7a712
May 26 11:24:29.064432 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.064414    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:29.064622 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:29.064605    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2dae63_45e0_481d_b9de_7eea78edbc6b.slice/crio-dbf34d54ff82df590d9632f486929b2696e96386df10f5e1782cce992ab7adf1 WatchSource:0}: Error finding container dbf34d54ff82df590d9632f486929b2696e96386df10f5e1782cce992ab7adf1: Status 404 returned error can't find the container with id dbf34d54ff82df590d9632f486929b2696e96386df10f5e1782cce992ab7adf1
May 26 11:24:29.069636 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.069613    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8dl7t"
May 26 11:24:29.070586 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:29.070568    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d1b0b3_d529_499b_8126_e90cbc5a8ac4.slice/crio-a55720684ac232b2707f60baa016ec88544d1c9d664e58b38888f50d82d266f1 WatchSource:0}: Error finding container a55720684ac232b2707f60baa016ec88544d1c9d664e58b38888f50d82d266f1: Status 404 returned error can't find the container with id a55720684ac232b2707f60baa016ec88544d1c9d664e58b38888f50d82d266f1
May 26 11:24:29.076236 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:29.076218    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1061f872_9b39_430c_9fe2_e1308e621e42.slice/crio-02d93bf1a5177b406699eba1f21aab14cd764513ff1496f6c160395b622bcfa0 WatchSource:0}: Error finding container 02d93bf1a5177b406699eba1f21aab14cd764513ff1496f6c160395b622bcfa0: Status 404 returned error can't find the container with id 02d93bf1a5177b406699eba1f21aab14cd764513ff1496f6c160395b622bcfa0
May 26 11:24:29.175675 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.175597    2563 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:29.273238 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.272722    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs\") pod \"network-metrics-daemon-vgwsb\" (UID: \"a9201d70-b657-4adf-9d01-5363b9fb7ded\") " pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:29.273238 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:29.272856    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:29.273238 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:29.272913    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs podName:a9201d70-b657-4adf-9d01-5363b9fb7ded nodeName:}" failed. No retries permitted until 2026-05-26 11:24:30.272894934 +0000 UTC m=+3.078733380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs") pod "network-metrics-daemon-vgwsb" (UID: "a9201d70-b657-4adf-9d01-5363b9fb7ded") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:29.373084 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.373053    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-52sp9\" (UniqueName: \"kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9\") pod \"network-check-target-5pwxn\" (UID: \"44da6752-3b21-42e6-942c-ab7fc44713dc\") " pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:29.373237 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.373115    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret\") pod \"global-pull-secret-syncer-wzshh\" (UID: \"7f39775e-7a53-44fb-8e18-70fd6a5764c5\") " pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:29.373329 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:29.373262    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:24:29.373388 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:29.373344    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret podName:7f39775e-7a53-44fb-8e18-70fd6a5764c5 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:30.373322943 +0000 UTC m=+3.179161398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret") pod "global-pull-secret-syncer-wzshh" (UID: "7f39775e-7a53-44fb-8e18-70fd6a5764c5") : object "kube-system"/"original-pull-secret" not registered
May 26 11:24:29.373753 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:29.373729    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:29.373753 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:29.373754    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:29.373909 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:29.373767    2563 projected.go:194] Error preparing data for projected volume kube-api-access-52sp9 for pod openshift-network-diagnostics/network-check-target-5pwxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:29.373909 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:29.373806    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9 podName:44da6752-3b21-42e6-942c-ab7fc44713dc nodeName:}" failed. No retries permitted until 2026-05-26 11:24:30.373793215 +0000 UTC m=+3.179631658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-52sp9" (UniqueName: "kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9") pod "network-check-target-5pwxn" (UID: "44da6752-3b21-42e6-942c-ab7fc44713dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:29.708982 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.708943    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 11:19:28 +0000 UTC" deadline="2028-01-18 09:18:12.373893949 +0000 UTC"
May 26 11:24:29.708982 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.708979    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14445h53m42.664917899s"
May 26 11:24:29.798758 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.798702    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8dl7t" event={"ID":"1061f872-9b39-430c-9fe2-e1308e621e42","Type":"ContainerStarted","Data":"02d93bf1a5177b406699eba1f21aab14cd764513ff1496f6c160395b622bcfa0"}
May 26 11:24:29.809671 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.809634    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gv6w6" event={"ID":"1c2dae63-45e0-481d-b9de-7eea78edbc6b","Type":"ContainerStarted","Data":"dbf34d54ff82df590d9632f486929b2696e96386df10f5e1782cce992ab7adf1"}
May 26 11:24:29.814887 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.814838    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rmwg" event={"ID":"daf3e280-5478-4519-83a2-5e5a7af26aa4","Type":"ContainerStarted","Data":"9db80ec38ca7dc11577cfcc2f24bb50a75a6d8c6f75ba7f7b3188051dbd7a712"}
May 26 11:24:29.819459 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.819376    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fhpf8" event={"ID":"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8","Type":"ContainerStarted","Data":"baa9edf1c2ee51a7e598f9481ad564322f38f4ec200ff8dbd471360d5f1fb65a"}
May 26 11:24:29.834866 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.834810    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zk6bl" event={"ID":"bd8ad6b4-eb8d-41fe-8247-de669518e6d4","Type":"ContainerStarted","Data":"57261f91f2184324f7350dbd6b62fcbc111ca51666ec246d3ce151e52025448e"}
May 26 11:24:29.838273 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.838248    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl" event={"ID":"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4","Type":"ContainerStarted","Data":"a55720684ac232b2707f60baa016ec88544d1c9d664e58b38888f50d82d266f1"}
May 26 11:24:29.853577 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.853501    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s4hrh" event={"ID":"7a6b09f3-e909-4c40-a22c-a6dd5beab731","Type":"ContainerStarted","Data":"7e46c237ab6b86d923111f1801a36133ff0a4a30a422e6cdff1254f2ac1f1529"}
May 26 11:24:29.865681 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.865653    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx" event={"ID":"31245d18-ee73-4dd5-9f1e-e0fe150f05d9","Type":"ContainerStarted","Data":"936bd463545257dcdf87300c37006b2341f5f7c44902e227b31e380b9e692367"}
May 26 11:24:29.985319 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:29.985238    2563 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:30.078933 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:30.078903    2563 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:30.280833 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:30.280744    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs\") pod \"network-metrics-daemon-vgwsb\" (UID: \"a9201d70-b657-4adf-9d01-5363b9fb7ded\") " pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:30.281000 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:30.280924    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:30.281000 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:30.280996    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs podName:a9201d70-b657-4adf-9d01-5363b9fb7ded nodeName:}" failed. No retries permitted until 2026-05-26 11:24:32.280970598 +0000 UTC m=+5.086809052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs") pod "network-metrics-daemon-vgwsb" (UID: "a9201d70-b657-4adf-9d01-5363b9fb7ded") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:30.382125 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:30.382083    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-52sp9\" (UniqueName: \"kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9\") pod \"network-check-target-5pwxn\" (UID: \"44da6752-3b21-42e6-942c-ab7fc44713dc\") " pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:30.382315 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:30.382165    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret\") pod \"global-pull-secret-syncer-wzshh\" (UID: \"7f39775e-7a53-44fb-8e18-70fd6a5764c5\") " pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:30.382410 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:30.382330    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:24:30.382410 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:30.382396    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret podName:7f39775e-7a53-44fb-8e18-70fd6a5764c5 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:32.382376441 +0000 UTC m=+5.188214884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret") pod "global-pull-secret-syncer-wzshh" (UID: "7f39775e-7a53-44fb-8e18-70fd6a5764c5") : object "kube-system"/"original-pull-secret" not registered
May 26 11:24:30.382834 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:30.382814    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:30.382917 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:30.382840    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:30.382917 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:30.382853    2563 projected.go:194] Error preparing data for projected volume kube-api-access-52sp9 for pod openshift-network-diagnostics/network-check-target-5pwxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:30.382917 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:30.382900    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9 podName:44da6752-3b21-42e6-942c-ab7fc44713dc nodeName:}" failed. No retries permitted until 2026-05-26 11:24:32.382884503 +0000 UTC m=+5.188722951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-52sp9" (UniqueName: "kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9") pod "network-check-target-5pwxn" (UID: "44da6752-3b21-42e6-942c-ab7fc44713dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:30.709791 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:30.709701    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 11:19:28 +0000 UTC" deadline="2028-01-06 01:22:15.103472032 +0000 UTC"
May 26 11:24:30.709791 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:30.709733    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14149h57m44.393742225s"
May 26 11:24:30.792392 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:30.791461    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:30.792392 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:30.791603    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:24:30.792392 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:30.792060    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:30.792392 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:30.792152    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:30.792392 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:30.792225    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:30.792392 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:30.792343    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:32.297900 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:32.297296    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs\") pod \"network-metrics-daemon-vgwsb\" (UID: \"a9201d70-b657-4adf-9d01-5363b9fb7ded\") " pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:32.297900 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:32.297489    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:32.297900 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:32.297550    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs podName:a9201d70-b657-4adf-9d01-5363b9fb7ded nodeName:}" failed. No retries permitted until 2026-05-26 11:24:36.297531344 +0000 UTC m=+9.103369787 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs") pod "network-metrics-daemon-vgwsb" (UID: "a9201d70-b657-4adf-9d01-5363b9fb7ded") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:32.399066 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:32.398477    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-52sp9\" (UniqueName: \"kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9\") pod \"network-check-target-5pwxn\" (UID: \"44da6752-3b21-42e6-942c-ab7fc44713dc\") " pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:32.399066 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:32.398560    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret\") pod \"global-pull-secret-syncer-wzshh\" (UID: \"7f39775e-7a53-44fb-8e18-70fd6a5764c5\") " pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:32.399066 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:32.398680    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:24:32.399066 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:32.398731    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret podName:7f39775e-7a53-44fb-8e18-70fd6a5764c5 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:36.398715772 +0000 UTC m=+9.204554223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret") pod "global-pull-secret-syncer-wzshh" (UID: "7f39775e-7a53-44fb-8e18-70fd6a5764c5") : object "kube-system"/"original-pull-secret" not registered
May 26 11:24:32.399066 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:32.398775    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:32.399066 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:32.398800    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:32.399066 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:32.398813    2563 projected.go:194] Error preparing data for projected volume kube-api-access-52sp9 for pod openshift-network-diagnostics/network-check-target-5pwxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:32.399066 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:32.398863    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9 podName:44da6752-3b21-42e6-942c-ab7fc44713dc nodeName:}" failed. No retries permitted until 2026-05-26 11:24:36.398846767 +0000 UTC m=+9.204685213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-52sp9" (UniqueName: "kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9") pod "network-check-target-5pwxn" (UID: "44da6752-3b21-42e6-942c-ab7fc44713dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:32.792810 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:32.792256    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:32.792810 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:32.792402    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:32.792810 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:32.792406    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:32.792810 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:32.792408    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:32.792810 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:32.792519    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:32.792810 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:32.792624    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:24:34.792005 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:34.791938    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:34.792005 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:34.791964    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:34.792558 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:34.791938    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:34.792558 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:34.792090    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:24:34.792558 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:34.792213    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:34.792862 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:34.792827    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:36.330261 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:36.329828    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs\") pod \"network-metrics-daemon-vgwsb\" (UID: \"a9201d70-b657-4adf-9d01-5363b9fb7ded\") " pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:36.330261 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:36.329954    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:36.330261 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:36.330001    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs podName:a9201d70-b657-4adf-9d01-5363b9fb7ded nodeName:}" failed. No retries permitted until 2026-05-26 11:24:44.329987686 +0000 UTC m=+17.135826128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs") pod "network-metrics-daemon-vgwsb" (UID: "a9201d70-b657-4adf-9d01-5363b9fb7ded") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:36.431202 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:36.430358    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-52sp9\" (UniqueName: \"kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9\") pod \"network-check-target-5pwxn\" (UID: \"44da6752-3b21-42e6-942c-ab7fc44713dc\") " pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:36.431202 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:36.430435    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret\") pod \"global-pull-secret-syncer-wzshh\" (UID: \"7f39775e-7a53-44fb-8e18-70fd6a5764c5\") " pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:36.431202 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:36.430583    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:24:36.431202 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:36.430651    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret podName:7f39775e-7a53-44fb-8e18-70fd6a5764c5 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:44.430631218 +0000 UTC m=+17.236469665 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret") pod "global-pull-secret-syncer-wzshh" (UID: "7f39775e-7a53-44fb-8e18-70fd6a5764c5") : object "kube-system"/"original-pull-secret" not registered
May 26 11:24:36.431202 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:36.431084    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:36.431202 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:36.431104    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:36.431202 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:36.431117    2563 projected.go:194] Error preparing data for projected volume kube-api-access-52sp9 for pod openshift-network-diagnostics/network-check-target-5pwxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:36.431202 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:36.431159    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9 podName:44da6752-3b21-42e6-942c-ab7fc44713dc nodeName:}" failed. No retries permitted until 2026-05-26 11:24:44.431145388 +0000 UTC m=+17.236983830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-52sp9" (UniqueName: "kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9") pod "network-check-target-5pwxn" (UID: "44da6752-3b21-42e6-942c-ab7fc44713dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:36.792276 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:36.792167    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:36.792276 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:36.792189    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:36.792276 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:36.792167    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:36.792551 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:36.792318    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:24:36.792852 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:36.792717    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:36.792852 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:36.792806    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:38.791578 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:38.791533    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:38.791578 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:38.791578    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:38.792027 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:38.791669    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:38.792027 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:38.791715    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:38.792027 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:38.791753    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:38.792027 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:38.791829    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:24:40.792159 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:40.792120    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:40.792582 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:40.792169    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:40.792582 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:40.792119    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:40.792582 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:40.792229    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:40.792582 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:40.792319    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:24:40.792582 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:40.792377    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:42.791784 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:42.791755    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:42.791784 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:42.791771    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:42.792342 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:42.791759    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:42.792342 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:42.791847    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:42.792342 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:42.791921    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:42.792342 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:42.792011    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:24:44.391726 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:44.391689    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs\") pod \"network-metrics-daemon-vgwsb\" (UID: \"a9201d70-b657-4adf-9d01-5363b9fb7ded\") " pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:44.392239 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:44.391886    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:44.392239 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:44.391974    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs podName:a9201d70-b657-4adf-9d01-5363b9fb7ded nodeName:}" failed. No retries permitted until 2026-05-26 11:25:00.391950261 +0000 UTC m=+33.197788711 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs") pod "network-metrics-daemon-vgwsb" (UID: "a9201d70-b657-4adf-9d01-5363b9fb7ded") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:44.492136 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:44.492090    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-52sp9\" (UniqueName: \"kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9\") pod \"network-check-target-5pwxn\" (UID: \"44da6752-3b21-42e6-942c-ab7fc44713dc\") " pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:44.492338 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:44.492161    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret\") pod \"global-pull-secret-syncer-wzshh\" (UID: \"7f39775e-7a53-44fb-8e18-70fd6a5764c5\") " pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:44.492338 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:44.492247    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:24:44.492338 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:44.492262    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:44.492338 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:44.492296    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:44.492338 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:44.492310    2563 projected.go:194] Error preparing data for projected volume kube-api-access-52sp9 for pod openshift-network-diagnostics/network-check-target-5pwxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:44.492338 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:44.492330    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret podName:7f39775e-7a53-44fb-8e18-70fd6a5764c5 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:00.492311805 +0000 UTC m=+33.298150261 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret") pod "global-pull-secret-syncer-wzshh" (UID: "7f39775e-7a53-44fb-8e18-70fd6a5764c5") : object "kube-system"/"original-pull-secret" not registered
May 26 11:24:44.492595 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:44.492359    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9 podName:44da6752-3b21-42e6-942c-ab7fc44713dc nodeName:}" failed. No retries permitted until 2026-05-26 11:25:00.492349261 +0000 UTC m=+33.298187724 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-52sp9" (UniqueName: "kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9") pod "network-check-target-5pwxn" (UID: "44da6752-3b21-42e6-942c-ab7fc44713dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:44.791426 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:44.791381    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:44.791621 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:44.791435    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:44.791621 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:44.791493    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:44.791621 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:44.791506    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:44.791621 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:44.791571    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:44.791831 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:44.791655    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:24:46.791719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:46.791462    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:46.792126 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:46.791473    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:46.792126 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:46.791841    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:24:46.792126 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:46.791486    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:46.792126 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:46.791885    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:46.792126 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:46.791950    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:48.791248 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.791221    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:48.791946 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.791225    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:48.791946 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:48.791353    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:48.791946 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.791223    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:48.791946 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:48.791415    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:24:48.791946 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:48.791448    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:48.900683 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.900643    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8dl7t" event={"ID":"1061f872-9b39-430c-9fe2-e1308e621e42","Type":"ContainerStarted","Data":"54ca626d59299319078a253f7032a42d2ee564a046424b4fc6c7d0316af903aa"}
May 26 11:24:48.902309 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.902246    2563 generic.go:358] "Generic (PLEG): container finished" podID="daf3e280-5478-4519-83a2-5e5a7af26aa4" containerID="b3dde53c9ae26119198c98933dbd4b41f1e443c56e17445196510616ba902bf4" exitCode=0
May 26 11:24:48.902440 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.902367    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rmwg" event={"ID":"daf3e280-5478-4519-83a2-5e5a7af26aa4","Type":"ContainerDied","Data":"b3dde53c9ae26119198c98933dbd4b41f1e443c56e17445196510616ba902bf4"}
May 26 11:24:48.904018 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.903975    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fhpf8" event={"ID":"3b46a615-dbda-42ba-ba5b-0ac8bb3076c8","Type":"ContainerStarted","Data":"a806032e9a49cf67900860095d72220deb616d843364c03b104a0cc46ef00b35"}
May 26 11:24:48.905729 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.905708    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zk6bl" event={"ID":"bd8ad6b4-eb8d-41fe-8247-de669518e6d4","Type":"ContainerStarted","Data":"e3ef62a9a1b2065dfda6d58a525021b7da2c7bbddba89bc2224003b49857c396"}
May 26 11:24:48.909581 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.909558    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl" event={"ID":"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4","Type":"ContainerStarted","Data":"9daf66fe4add49e5399b75d8ea9de7011d9b93d1edc24d3a646375f714f17038"}
May 26 11:24:48.909663 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.909591    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl" event={"ID":"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4","Type":"ContainerStarted","Data":"6166a8ffdfa7582e7e4055ec498e5ed609339f13c325a93029da33027026f84a"}
May 26 11:24:48.909663 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.909610    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl" event={"ID":"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4","Type":"ContainerStarted","Data":"390f32e09011b39616b43cc3dcf0b5a4a8281dc9ad613ef38a8fad80f97968d9"}
May 26 11:24:48.909663 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.909623    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl" event={"ID":"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4","Type":"ContainerStarted","Data":"01864167fbde59733c2ec6063b93b07e1bc24cc4f801289ca35d97ae5f9fe39e"}
May 26 11:24:48.909663 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.909635    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl" event={"ID":"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4","Type":"ContainerStarted","Data":"cddd61bcd386d3220cdbf2df8be684b9592211ee1823a38c1cb50cdc0427ac2a"}
May 26 11:24:48.909663 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.909647    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl" event={"ID":"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4","Type":"ContainerStarted","Data":"5426236ebc7b1e828aad2abdff4717ef440384aa61826699ac02fda59158ba48"}
May 26 11:24:48.911402 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.911371    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s4hrh" event={"ID":"7a6b09f3-e909-4c40-a22c-a6dd5beab731","Type":"ContainerStarted","Data":"0259c4b07b891993f26a102b5e7a06a99b039f5fb6ab068bb7dc3d5924836817"}
May 26 11:24:48.912881 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.912855    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx" event={"ID":"31245d18-ee73-4dd5-9f1e-e0fe150f05d9","Type":"ContainerStarted","Data":"bf07cdfd2b7593b6f1853274e6c084c57f0dfb8c42c98ff7d54692dd084da35b"}
May 26 11:24:48.914414 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.914382    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal" event={"ID":"9a50d4f711fbf255cb7babbc8960a87b","Type":"ContainerStarted","Data":"ed2320637963df5c644570b2917e1f667c116a3068933ef4e6528ce449646c44"}
May 26 11:24:48.915935 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.915892    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8dl7t" podStartSLOduration=3.264548648 podStartE2EDuration="21.915881769s" podCreationTimestamp="2026-05-26 11:24:27 +0000 UTC" firstStartedPulling="2026-05-26 11:24:29.077964779 +0000 UTC m=+1.883803224" lastFinishedPulling="2026-05-26 11:24:47.729297904 +0000 UTC m=+20.535136345" observedRunningTime="2026-05-26 11:24:48.914972862 +0000 UTC m=+21.720811330" watchObservedRunningTime="2026-05-26 11:24:48.915881769 +0000 UTC m=+21.721720233"
May 26 11:24:48.917039 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.917014    2563 generic.go:358] "Generic (PLEG): container finished" podID="dcd208f0122d6cb3e226ed802a3b1ab4" containerID="ffeb127b8916ddc00ed83b2466bda92f9fa1eafd710e466590dbc9a1732c30a6" exitCode=0
May 26 11:24:48.917143 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.917052    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal" event={"ID":"dcd208f0122d6cb3e226ed802a3b1ab4","Type":"ContainerDied","Data":"ffeb127b8916ddc00ed83b2466bda92f9fa1eafd710e466590dbc9a1732c30a6"}
May 26 11:24:48.929037 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.928991    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zk6bl" podStartSLOduration=3.186412408 podStartE2EDuration="21.928976822s" podCreationTimestamp="2026-05-26 11:24:27 +0000 UTC" firstStartedPulling="2026-05-26 11:24:28.988019033 +0000 UTC m=+1.793857478" lastFinishedPulling="2026-05-26 11:24:47.730583447 +0000 UTC m=+20.536421892" observedRunningTime="2026-05-26 11:24:48.928872475 +0000 UTC m=+21.734710953" watchObservedRunningTime="2026-05-26 11:24:48.928976822 +0000 UTC m=+21.734815283"
May 26 11:24:48.961992 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.961574    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s4hrh" podStartSLOduration=3.242489425 podStartE2EDuration="21.961557351s" podCreationTimestamp="2026-05-26 11:24:27 +0000 UTC" firstStartedPulling="2026-05-26 11:24:29.011628148 +0000 UTC m=+1.817466594" lastFinishedPulling="2026-05-26 11:24:47.730696064 +0000 UTC m=+20.536534520" observedRunningTime="2026-05-26 11:24:48.944701221 +0000 UTC m=+21.750539688" watchObservedRunningTime="2026-05-26 11:24:48.961557351 +0000 UTC m=+21.767395815"
May 26 11:24:48.976342 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.976241    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-71.ec2.internal" podStartSLOduration=20.976226496 podStartE2EDuration="20.976226496s" podCreationTimestamp="2026-05-26 11:24:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:24:48.97611596 +0000 UTC m=+21.781954428" watchObservedRunningTime="2026-05-26 11:24:48.976226496 +0000 UTC m=+21.782064963"
May 26 11:24:48.977315 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:48.976758    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-fhpf8" podStartSLOduration=3.292549556 podStartE2EDuration="21.976743462s" podCreationTimestamp="2026-05-26 11:24:27 +0000 UTC" firstStartedPulling="2026-05-26 11:24:29.057561153 +0000 UTC m=+1.863399596" lastFinishedPulling="2026-05-26 11:24:47.74175506 +0000 UTC m=+20.547593502" observedRunningTime="2026-05-26 11:24:48.961830893 +0000 UTC m=+21.767669359" watchObservedRunningTime="2026-05-26 11:24:48.976743462 +0000 UTC m=+21.782581928"
May 26 11:24:49.269245 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:49.269221    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 26 11:24:49.729623 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:49.729480    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-26T11:24:49.269241436Z","UUID":"e41acbc8-a55d-4401-a3c3-14020045c1ef","Handler":null,"Name":"","Endpoint":""}
May 26 11:24:49.732373 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:49.732345    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 26 11:24:49.732509 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:49.732385    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 26 11:24:49.921617 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:49.921545    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal" event={"ID":"dcd208f0122d6cb3e226ed802a3b1ab4","Type":"ContainerStarted","Data":"66a9696388b29617b785c33421618889dc44843884cd33f408ccfea79be558e6"}
May 26 11:24:49.923120 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:49.923090    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gv6w6" event={"ID":"1c2dae63-45e0-481d-b9de-7eea78edbc6b","Type":"ContainerStarted","Data":"8a70d65076aa715b7ca9435871b93a6c13a7f8b699b1337d5c09eaceb76f0b77"}
May 26 11:24:49.925535 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:49.925341    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx" event={"ID":"31245d18-ee73-4dd5-9f1e-e0fe150f05d9","Type":"ContainerStarted","Data":"bf256bd3bfcf44adb8a39fb5f0f81f3a0b671eacce26f87a22502c62fab4f90e"}
May 26 11:24:49.937995 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:49.937953    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-71.ec2.internal" podStartSLOduration=21.937941823 podStartE2EDuration="21.937941823s" podCreationTimestamp="2026-05-26 11:24:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:24:49.937926198 +0000 UTC m=+22.743764721" watchObservedRunningTime="2026-05-26 11:24:49.937941823 +0000 UTC m=+22.743780307"
May 26 11:24:49.951792 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:49.951748    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gv6w6" podStartSLOduration=4.288864735 podStartE2EDuration="22.951733694s" podCreationTimestamp="2026-05-26 11:24:27 +0000 UTC" firstStartedPulling="2026-05-26 11:24:29.066246183 +0000 UTC m=+1.872084624" lastFinishedPulling="2026-05-26 11:24:47.729115136 +0000 UTC m=+20.534953583" observedRunningTime="2026-05-26 11:24:49.951676415 +0000 UTC m=+22.757514880" watchObservedRunningTime="2026-05-26 11:24:49.951733694 +0000 UTC m=+22.757572166"
May 26 11:24:50.791853 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:50.791820    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:50.792036 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:50.791820    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:50.792036 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:50.791927    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:50.792036 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:50.791819    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:50.792036 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:50.791998    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:24:50.792204 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:50.792079    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:50.930214 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:50.929899    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx" event={"ID":"31245d18-ee73-4dd5-9f1e-e0fe150f05d9","Type":"ContainerStarted","Data":"fc34f1ca6a77a21e182186bda1c7ebebf99b21f56ae50676e66095790a326f2c"}
May 26 11:24:50.961498 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:50.961258    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5g7kx" podStartSLOduration=2.876929246 podStartE2EDuration="23.961243931s" podCreationTimestamp="2026-05-26 11:24:27 +0000 UTC" firstStartedPulling="2026-05-26 11:24:29.007318162 +0000 UTC m=+1.813156603" lastFinishedPulling="2026-05-26 11:24:50.091632842 +0000 UTC m=+22.897471288" observedRunningTime="2026-05-26 11:24:50.960196358 +0000 UTC m=+23.766034817" watchObservedRunningTime="2026-05-26 11:24:50.961243931 +0000 UTC m=+23.767082395"
May 26 11:24:51.256414 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.256374    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zk6bl"
May 26 11:24:51.257159 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.257134    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zk6bl"
May 26 11:24:51.369789 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.369763    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-nqs65"]
May 26 11:24:51.373189 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.373168    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nqs65"
May 26 11:24:51.375518 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.375493    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 26 11:24:51.375656 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.375635    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vzp6h\""
May 26 11:24:51.375656 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.375657    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 26 11:24:51.440640 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.440598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3e68b4-5329-4f10-bf42-4f7e16a2795c-tmp-dir\") pod \"node-resolver-nqs65\" (UID: \"9d3e68b4-5329-4f10-bf42-4f7e16a2795c\") " pod="openshift-dns/node-resolver-nqs65"
May 26 11:24:51.440801 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.440726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w74v\" (UniqueName: \"kubernetes.io/projected/9d3e68b4-5329-4f10-bf42-4f7e16a2795c-kube-api-access-8w74v\") pod \"node-resolver-nqs65\" (UID: \"9d3e68b4-5329-4f10-bf42-4f7e16a2795c\") " pod="openshift-dns/node-resolver-nqs65"
May 26 11:24:51.440801 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.440770    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9d3e68b4-5329-4f10-bf42-4f7e16a2795c-hosts-file\") pod \"node-resolver-nqs65\" (UID: \"9d3e68b4-5329-4f10-bf42-4f7e16a2795c\") " pod="openshift-dns/node-resolver-nqs65"
May 26 11:24:51.542157 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.542083    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3e68b4-5329-4f10-bf42-4f7e16a2795c-tmp-dir\") pod \"node-resolver-nqs65\" (UID: \"9d3e68b4-5329-4f10-bf42-4f7e16a2795c\") " pod="openshift-dns/node-resolver-nqs65"
May 26 11:24:51.542437 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.542294    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8w74v\" (UniqueName: \"kubernetes.io/projected/9d3e68b4-5329-4f10-bf42-4f7e16a2795c-kube-api-access-8w74v\") pod \"node-resolver-nqs65\" (UID: \"9d3e68b4-5329-4f10-bf42-4f7e16a2795c\") " pod="openshift-dns/node-resolver-nqs65"
May 26 11:24:51.542437 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.542343    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9d3e68b4-5329-4f10-bf42-4f7e16a2795c-hosts-file\") pod \"node-resolver-nqs65\" (UID: \"9d3e68b4-5329-4f10-bf42-4f7e16a2795c\") " pod="openshift-dns/node-resolver-nqs65"
May 26 11:24:51.542548 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.542439    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3e68b4-5329-4f10-bf42-4f7e16a2795c-tmp-dir\") pod \"node-resolver-nqs65\" (UID: \"9d3e68b4-5329-4f10-bf42-4f7e16a2795c\") " pod="openshift-dns/node-resolver-nqs65"
May 26 11:24:51.542548 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.542446    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9d3e68b4-5329-4f10-bf42-4f7e16a2795c-hosts-file\") pod \"node-resolver-nqs65\" (UID: \"9d3e68b4-5329-4f10-bf42-4f7e16a2795c\") " pod="openshift-dns/node-resolver-nqs65"
May 26 11:24:51.555297 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.555265    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w74v\" (UniqueName: \"kubernetes.io/projected/9d3e68b4-5329-4f10-bf42-4f7e16a2795c-kube-api-access-8w74v\") pod \"node-resolver-nqs65\" (UID: \"9d3e68b4-5329-4f10-bf42-4f7e16a2795c\") " pod="openshift-dns/node-resolver-nqs65"
May 26 11:24:51.683683 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.683650    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nqs65"
May 26 11:24:51.934194 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:51.934101    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl" event={"ID":"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4","Type":"ContainerStarted","Data":"66c3cf889137d6a4cbad6916d98f408753e60b2ba759baa5562d571caff378b0"}
May 26 11:24:52.792148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:52.792101    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:52.792148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:52.792131    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:52.792447 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:52.792254    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:24:52.792447 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:52.792298    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:52.792447 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:52.792392    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:52.792564 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:52.792437    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:52.814659 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:24:52.814630    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3e68b4_5329_4f10_bf42_4f7e16a2795c.slice/crio-2252a09726d9f6b8f102a8020e55b996bd726bd82740469c24d2a39b26cb94a3 WatchSource:0}: Error finding container 2252a09726d9f6b8f102a8020e55b996bd726bd82740469c24d2a39b26cb94a3: Status 404 returned error can't find the container with id 2252a09726d9f6b8f102a8020e55b996bd726bd82740469c24d2a39b26cb94a3
May 26 11:24:52.936614 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:52.936586    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nqs65" event={"ID":"9d3e68b4-5329-4f10-bf42-4f7e16a2795c","Type":"ContainerStarted","Data":"2252a09726d9f6b8f102a8020e55b996bd726bd82740469c24d2a39b26cb94a3"}
May 26 11:24:53.323613 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:53.323329    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zk6bl"
May 26 11:24:53.323760 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:53.323645    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 26 11:24:53.324079 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:53.324064    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zk6bl"
May 26 11:24:53.940773 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:53.940741    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl" event={"ID":"a1d1b0b3-d529-499b-8126-e90cbc5a8ac4","Type":"ContainerStarted","Data":"5b1e26e78273f40683f5d5b4ea1c23c0657e41ac2e5aacd203af35b58ad76038"}
May 26 11:24:53.941508 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:53.940992    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:53.941508 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:53.941013    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:53.945617 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:53.945592    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nqs65" event={"ID":"9d3e68b4-5329-4f10-bf42-4f7e16a2795c","Type":"ContainerStarted","Data":"69632c0f4a6cc33f19a9d834dbf82f34bae177e0122afbd6ccaedc849f7ac181"}
May 26 11:24:53.947270 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:53.947247    2563 generic.go:358] "Generic (PLEG): container finished" podID="daf3e280-5478-4519-83a2-5e5a7af26aa4" containerID="49a262f56621cd4d9049542fb63a42b167087dca6ec5e868950e4fa3bbd89032" exitCode=0
May 26 11:24:53.947422 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:53.947317    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rmwg" event={"ID":"daf3e280-5478-4519-83a2-5e5a7af26aa4","Type":"ContainerDied","Data":"49a262f56621cd4d9049542fb63a42b167087dca6ec5e868950e4fa3bbd89032"}
May 26 11:24:53.957667 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:53.957644    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:53.972215 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:53.972179    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl" podStartSLOduration=7.851349268 podStartE2EDuration="26.972168193s" podCreationTimestamp="2026-05-26 11:24:27 +0000 UTC" firstStartedPulling="2026-05-26 11:24:29.072565762 +0000 UTC m=+1.878404204" lastFinishedPulling="2026-05-26 11:24:48.193384672 +0000 UTC m=+20.999223129" observedRunningTime="2026-05-26 11:24:53.971683852 +0000 UTC m=+26.777522315" watchObservedRunningTime="2026-05-26 11:24:53.972168193 +0000 UTC m=+26.778006656"
May 26 11:24:54.035073 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:54.035027    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-nqs65" podStartSLOduration=3.03501575 podStartE2EDuration="3.03501575s" podCreationTimestamp="2026-05-26 11:24:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:24:54.034529345 +0000 UTC m=+26.840367809" watchObservedRunningTime="2026-05-26 11:24:54.03501575 +0000 UTC m=+26.840854213"
May 26 11:24:54.791817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:54.791677    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:54.791957 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:54.791677    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:54.791957 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:54.791898    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:54.792064 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:54.792026    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:24:54.792064 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:54.791680    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:54.792169 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:54.792127    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:54.951438 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:54.951323    2563 generic.go:358] "Generic (PLEG): container finished" podID="daf3e280-5478-4519-83a2-5e5a7af26aa4" containerID="fde10a2aea6ab454aefb77ffab65e2ed6d74208d9fc7682b10462c7ea0c30705" exitCode=0
May 26 11:24:54.951438 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:54.951415    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rmwg" event={"ID":"daf3e280-5478-4519-83a2-5e5a7af26aa4","Type":"ContainerDied","Data":"fde10a2aea6ab454aefb77ffab65e2ed6d74208d9fc7682b10462c7ea0c30705"}
May 26 11:24:54.952726 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:54.952213    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:54.968350 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:54.968331    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:24:54.992032 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:54.992007    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wzshh"]
May 26 11:24:54.992147 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:54.992114    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:54.992244 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:54.992223    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:54.998857 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:54.998835    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vgwsb"]
May 26 11:24:54.998951 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:54.998923    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:54.999006 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:54.998990    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:24:55.010215 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:55.010195    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5pwxn"]
May 26 11:24:55.010316 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:55.010269    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:55.010364 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:55.010339    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:55.955173 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:55.955081    2563 generic.go:358] "Generic (PLEG): container finished" podID="daf3e280-5478-4519-83a2-5e5a7af26aa4" containerID="282ad4b985ab04d33c14ef06ebec02ca579378090ce49123e7fc2bdb2e92fc6a" exitCode=0
May 26 11:24:55.955173 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:55.955122    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rmwg" event={"ID":"daf3e280-5478-4519-83a2-5e5a7af26aa4","Type":"ContainerDied","Data":"282ad4b985ab04d33c14ef06ebec02ca579378090ce49123e7fc2bdb2e92fc6a"}
May 26 11:24:56.791466 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:56.791436    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:56.791631 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:56.791541    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:56.791631 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:56.791552    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:56.791631 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:56.791617    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:56.791631 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:56.791627    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:56.791795 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:56.791724    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:24:58.791566 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:58.791531    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:24:58.791946 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:58.791631    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:24:58.791946 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:58.791657    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:24:58.791946 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:58.791747    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:24:58.791946 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:24:58.791784    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:24:58.791946 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:24:58.791838    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:25:00.414005 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:00.413964    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs\") pod \"network-metrics-daemon-vgwsb\" (UID: \"a9201d70-b657-4adf-9d01-5363b9fb7ded\") " pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:25:00.414490 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:00.414132    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:00.414490 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:00.414214    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs podName:a9201d70-b657-4adf-9d01-5363b9fb7ded nodeName:}" failed. No retries permitted until 2026-05-26 11:25:32.414193383 +0000 UTC m=+65.220031825 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs") pod "network-metrics-daemon-vgwsb" (UID: "a9201d70-b657-4adf-9d01-5363b9fb7ded") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:00.514942 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:00.514908    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret\") pod \"global-pull-secret-syncer-wzshh\" (UID: \"7f39775e-7a53-44fb-8e18-70fd6a5764c5\") " pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:25:00.515097 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:00.514982    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-52sp9\" (UniqueName: \"kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9\") pod \"network-check-target-5pwxn\" (UID: \"44da6752-3b21-42e6-942c-ab7fc44713dc\") " pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:25:00.515097 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:00.515079    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:25:00.515185 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:00.515106    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:25:00.515185 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:00.515122    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:25:00.515185 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:00.515133    2563 projected.go:194] Error preparing data for projected volume kube-api-access-52sp9 for pod openshift-network-diagnostics/network-check-target-5pwxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:00.515185 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:00.515156    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret podName:7f39775e-7a53-44fb-8e18-70fd6a5764c5 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:32.51513607 +0000 UTC m=+65.320974516 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret") pod "global-pull-secret-syncer-wzshh" (UID: "7f39775e-7a53-44fb-8e18-70fd6a5764c5") : object "kube-system"/"original-pull-secret" not registered
May 26 11:25:00.515185 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:00.515176    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9 podName:44da6752-3b21-42e6-942c-ab7fc44713dc nodeName:}" failed. No retries permitted until 2026-05-26 11:25:32.515166268 +0000 UTC m=+65.321004717 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-52sp9" (UniqueName: "kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9") pod "network-check-target-5pwxn" (UID: "44da6752-3b21-42e6-942c-ab7fc44713dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:00.791607 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:00.791568    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:25:00.791607 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:00.791606    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:25:00.791876 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:00.791715    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgwsb" podUID="a9201d70-b657-4adf-9d01-5363b9fb7ded"
May 26 11:25:00.791876 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:00.791775    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pwxn" podUID="44da6752-3b21-42e6-942c-ab7fc44713dc"
May 26 11:25:00.791876 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:00.791824    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:25:00.792038 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:00.791905    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzshh" podUID="7f39775e-7a53-44fb-8e18-70fd6a5764c5"
May 26 11:25:01.979710 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:01.979631    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-71.ec2.internal" event="NodeReady"
May 26 11:25:01.980606 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:01.979795    2563 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 26 11:25:02.027497 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.027445    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74c679c6fb-mpsq2"]
May 26 11:25:02.052406 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.052366    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5chfj"]
May 26 11:25:02.052564 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.052533    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.055641 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.055615    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 26 11:25:02.055755 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.055625    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 26 11:25:02.055755 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.055695    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 26 11:25:02.057528 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.057508    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9tkg9\""
May 26 11:25:02.069603 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.069582    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rtrlh"]
May 26 11:25:02.069771 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.069752    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5chfj"
May 26 11:25:02.069976 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.069959    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 26 11:25:02.072429 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.072413    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 26 11:25:02.073151 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.073132    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-46vjq\""
May 26 11:25:02.073221 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.073135    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 26 11:25:02.092263 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.092240    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74c679c6fb-mpsq2"]
May 26 11:25:02.092263 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.092267    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5chfj"]
May 26 11:25:02.092396 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.092276    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rtrlh"]
May 26 11:25:02.092396 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.092374    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rtrlh"
May 26 11:25:02.096504 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.096488    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qmmns\""
May 26 11:25:02.096622 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.096534    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 26 11:25:02.097030 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.096974    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 26 11:25:02.097030 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.096986    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 26 11:25:02.227711 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.227681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/195ce04c-023e-47d6-b5f5-22b90196e4b2-ca-trust-extracted\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.227890 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.227731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/195ce04c-023e-47d6-b5f5-22b90196e4b2-installation-pull-secrets\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.227890 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.227799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pkk4\" (UniqueName: \"kubernetes.io/projected/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-kube-api-access-8pkk4\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:02.227890 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.227827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.227890 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.227845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-bound-sa-token\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.227890 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.227860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7f7z\" (UniqueName: \"kubernetes.io/projected/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-kube-api-access-p7f7z\") pod \"ingress-canary-rtrlh\" (UID: \"3f9d7bb9-08d8-495f-9713-a6432e7f6a4f\") " pod="openshift-ingress-canary/ingress-canary-rtrlh"
May 26 11:25:02.227890 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.227878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-certificates\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.228183 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.227897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-config-volume\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:02.228183 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.227917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-tmp-dir\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:02.228183 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.227963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert\") pod \"ingress-canary-rtrlh\" (UID: \"3f9d7bb9-08d8-495f-9713-a6432e7f6a4f\") " pod="openshift-ingress-canary/ingress-canary-rtrlh"
May 26 11:25:02.228183 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.228004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/195ce04c-023e-47d6-b5f5-22b90196e4b2-image-registry-private-configuration\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.228183 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.228033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/195ce04c-023e-47d6-b5f5-22b90196e4b2-trusted-ca\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.228183 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.228094    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:02.228183 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.228120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb8ps\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-kube-api-access-pb8ps\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.328496 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.328415    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:02.328496 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.328460    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pb8ps\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-kube-api-access-pb8ps\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.328496 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.328494    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/195ce04c-023e-47d6-b5f5-22b90196e4b2-ca-trust-extracted\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.328767 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:02.328598    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:02.328767 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.328662    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/195ce04c-023e-47d6-b5f5-22b90196e4b2-installation-pull-secrets\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.328767 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:02.328699    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls podName:e73ee593-f6c2-4e31-9297-0c2c0b321c4a nodeName:}" failed. No retries permitted until 2026-05-26 11:25:02.828676647 +0000 UTC m=+35.634515106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls") pod "dns-default-5chfj" (UID: "e73ee593-f6c2-4e31-9297-0c2c0b321c4a") : secret "dns-default-metrics-tls" not found
May 26 11:25:02.328767 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.328747    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8pkk4\" (UniqueName: \"kubernetes.io/projected/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-kube-api-access-8pkk4\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:02.328976 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.328778    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.328976 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.328810    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-bound-sa-token\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.328976 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.328838    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p7f7z\" (UniqueName: \"kubernetes.io/projected/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-kube-api-access-p7f7z\") pod \"ingress-canary-rtrlh\" (UID: \"3f9d7bb9-08d8-495f-9713-a6432e7f6a4f\") " pod="openshift-ingress-canary/ingress-canary-rtrlh"
May 26 11:25:02.328976 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:02.328868    2563 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:02.328976 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.328871    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-certificates\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.328976 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:02.328883    2563 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74c679c6fb-mpsq2: secret "image-registry-tls" not found
May 26 11:25:02.328976 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.328908    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-config-volume\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:02.328976 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:02.328954    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls podName:195ce04c-023e-47d6-b5f5-22b90196e4b2 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:02.828934028 +0000 UTC m=+35.634772476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls") pod "image-registry-74c679c6fb-mpsq2" (UID: "195ce04c-023e-47d6-b5f5-22b90196e4b2") : secret "image-registry-tls" not found
May 26 11:25:02.328976 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.328977    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-tmp-dir\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:02.329427 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.329010    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert\") pod \"ingress-canary-rtrlh\" (UID: \"3f9d7bb9-08d8-495f-9713-a6432e7f6a4f\") " pod="openshift-ingress-canary/ingress-canary-rtrlh"
May 26 11:25:02.329427 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.329035    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/195ce04c-023e-47d6-b5f5-22b90196e4b2-image-registry-private-configuration\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.329427 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.329059    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/195ce04c-023e-47d6-b5f5-22b90196e4b2-trusted-ca\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.329427 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.329249    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/195ce04c-023e-47d6-b5f5-22b90196e4b2-ca-trust-extracted\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.329427 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:02.329269    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:02.329427 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:02.329390    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert podName:3f9d7bb9-08d8-495f-9713-a6432e7f6a4f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:02.82937111 +0000 UTC m=+35.635209578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert") pod "ingress-canary-rtrlh" (UID: "3f9d7bb9-08d8-495f-9713-a6432e7f6a4f") : secret "canary-serving-cert" not found
May 26 11:25:02.329710 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.329668    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-certificates\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.329710 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.329687    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-config-volume\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:02.329786 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.329724    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-tmp-dir\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:02.330465 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.330356    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/195ce04c-023e-47d6-b5f5-22b90196e4b2-trusted-ca\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.332865 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.332845    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/195ce04c-023e-47d6-b5f5-22b90196e4b2-image-registry-private-configuration\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.332865 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.332852    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/195ce04c-023e-47d6-b5f5-22b90196e4b2-installation-pull-secrets\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.354299 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.354265    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pkk4\" (UniqueName: \"kubernetes.io/projected/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-kube-api-access-8pkk4\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:02.354384 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.354341    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-bound-sa-token\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.354558 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.354536    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb8ps\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-kube-api-access-pb8ps\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.354625 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.354611    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7f7z\" (UniqueName: \"kubernetes.io/projected/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-kube-api-access-p7f7z\") pod \"ingress-canary-rtrlh\" (UID: \"3f9d7bb9-08d8-495f-9713-a6432e7f6a4f\") " pod="openshift-ingress-canary/ingress-canary-rtrlh"
May 26 11:25:02.791811 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.791777    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:25:02.791980 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.791777    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:25:02.792027 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.791777    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:25:02.796930 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.796909    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 11:25:02.797074 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.796927    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zzwv5\""
May 26 11:25:02.797163 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.797148    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 11:25:02.797700 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.797687    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 11:25:02.797824 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.797805    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 11:25:02.797871 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.797857    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xvpvn\""
May 26 11:25:02.833276 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.833248    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:02.833413 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.833313    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert\") pod \"ingress-canary-rtrlh\" (UID: \"3f9d7bb9-08d8-495f-9713-a6432e7f6a4f\") " pod="openshift-ingress-canary/ingress-canary-rtrlh"
May 26 11:25:02.833413 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.833374    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:02.833413 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:02.833393    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:02.833525 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:02.833464    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert podName:3f9d7bb9-08d8-495f-9713-a6432e7f6a4f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:03.83344637 +0000 UTC m=+36.639284812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert") pod "ingress-canary-rtrlh" (UID: "3f9d7bb9-08d8-495f-9713-a6432e7f6a4f") : secret "canary-serving-cert" not found
May 26 11:25:02.833525 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:02.833481    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:02.833525 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:02.833503    2563 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:02.833525 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:02.833518    2563 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74c679c6fb-mpsq2: secret "image-registry-tls" not found
May 26 11:25:02.833663 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:02.833540    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls podName:e73ee593-f6c2-4e31-9297-0c2c0b321c4a nodeName:}" failed. No retries permitted until 2026-05-26 11:25:03.833524602 +0000 UTC m=+36.639363064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls") pod "dns-default-5chfj" (UID: "e73ee593-f6c2-4e31-9297-0c2c0b321c4a") : secret "dns-default-metrics-tls" not found
May 26 11:25:02.833663 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:02.833563    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls podName:195ce04c-023e-47d6-b5f5-22b90196e4b2 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:03.833551391 +0000 UTC m=+36.639389847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls") pod "image-registry-74c679c6fb-mpsq2" (UID: "195ce04c-023e-47d6-b5f5-22b90196e4b2") : secret "image-registry-tls" not found
May 26 11:25:02.971044 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.970965    2563 generic.go:358] "Generic (PLEG): container finished" podID="daf3e280-5478-4519-83a2-5e5a7af26aa4" containerID="6b722fcef507c0661db87ee1acab5ec34487abac6417c996a7e18c8e72228587" exitCode=0
May 26 11:25:02.971044 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:02.971016    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rmwg" event={"ID":"daf3e280-5478-4519-83a2-5e5a7af26aa4","Type":"ContainerDied","Data":"6b722fcef507c0661db87ee1acab5ec34487abac6417c996a7e18c8e72228587"}
May 26 11:25:03.841374 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:03.841333    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert\") pod \"ingress-canary-rtrlh\" (UID: \"3f9d7bb9-08d8-495f-9713-a6432e7f6a4f\") " pod="openshift-ingress-canary/ingress-canary-rtrlh"
May 26 11:25:03.841983 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:03.841399    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:03.841983 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:03.841453    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:03.841983 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:03.841504    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:03.841983 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:03.841545    2563 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:03.841983 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:03.841555    2563 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74c679c6fb-mpsq2: secret "image-registry-tls" not found
May 26 11:25:03.841983 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:03.841566    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert podName:3f9d7bb9-08d8-495f-9713-a6432e7f6a4f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:05.841551194 +0000 UTC m=+38.647389636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert") pod "ingress-canary-rtrlh" (UID: "3f9d7bb9-08d8-495f-9713-a6432e7f6a4f") : secret "canary-serving-cert" not found
May 26 11:25:03.841983 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:03.841591    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls podName:195ce04c-023e-47d6-b5f5-22b90196e4b2 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:05.841580641 +0000 UTC m=+38.647419082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls") pod "image-registry-74c679c6fb-mpsq2" (UID: "195ce04c-023e-47d6-b5f5-22b90196e4b2") : secret "image-registry-tls" not found
May 26 11:25:03.841983 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:03.841546    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:03.841983 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:03.841615    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls podName:e73ee593-f6c2-4e31-9297-0c2c0b321c4a nodeName:}" failed. No retries permitted until 2026-05-26 11:25:05.841609318 +0000 UTC m=+38.647447760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls") pod "dns-default-5chfj" (UID: "e73ee593-f6c2-4e31-9297-0c2c0b321c4a") : secret "dns-default-metrics-tls" not found
May 26 11:25:03.975222 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:03.975179    2563 generic.go:358] "Generic (PLEG): container finished" podID="daf3e280-5478-4519-83a2-5e5a7af26aa4" containerID="177f78231c5e2d8d2359d36c69ec60e6b8790ac5dd3afd6cb008f9b532d44134" exitCode=0
May 26 11:25:03.975384 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:03.975255    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rmwg" event={"ID":"daf3e280-5478-4519-83a2-5e5a7af26aa4","Type":"ContainerDied","Data":"177f78231c5e2d8d2359d36c69ec60e6b8790ac5dd3afd6cb008f9b532d44134"}
May 26 11:25:04.980212 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:04.980182    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rmwg" event={"ID":"daf3e280-5478-4519-83a2-5e5a7af26aa4","Type":"ContainerStarted","Data":"e3e13c221195373f9578981ef12f0ea9f93316f50cfe44e31eacffb105deaaf5"}
May 26 11:25:05.009370 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:05.009319    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7rmwg" podStartSLOduration=4.689167945 podStartE2EDuration="38.009305033s" podCreationTimestamp="2026-05-26 11:24:27 +0000 UTC" firstStartedPulling="2026-05-26 11:24:29.061892579 +0000 UTC m=+1.867731021" lastFinishedPulling="2026-05-26 11:25:02.382029664 +0000 UTC m=+35.187868109" observedRunningTime="2026-05-26 11:25:05.007810903 +0000 UTC m=+37.813649367" watchObservedRunningTime="2026-05-26 11:25:05.009305033 +0000 UTC m=+37.815143494"
May 26 11:25:05.838253 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:05.838215    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-5z2cp"]
May 26 11:25:05.857123 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:05.857097    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-5z2cp"]
May 26 11:25:05.857292 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:05.857220    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-5z2cp"
May 26 11:25:05.857354 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:05.857268    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert\") pod \"ingress-canary-rtrlh\" (UID: \"3f9d7bb9-08d8-495f-9713-a6432e7f6a4f\") " pod="openshift-ingress-canary/ingress-canary-rtrlh"
May 26 11:25:05.857354 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:05.857326    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:05.857444 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:05.857402    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:05.857499 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:05.857439    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:05.857499 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:05.857494    2563 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:05.857591 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:05.857506    2563 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74c679c6fb-mpsq2: secret "image-registry-tls" not found
May 26 11:25:05.857591 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:05.857528    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert podName:3f9d7bb9-08d8-495f-9713-a6432e7f6a4f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:09.857507724 +0000 UTC m=+42.663346180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert") pod "ingress-canary-rtrlh" (UID: "3f9d7bb9-08d8-495f-9713-a6432e7f6a4f") : secret "canary-serving-cert" not found
May 26 11:25:05.857591 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:05.857547    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls podName:195ce04c-023e-47d6-b5f5-22b90196e4b2 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:09.857536745 +0000 UTC m=+42.663375187 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls") pod "image-registry-74c679c6fb-mpsq2" (UID: "195ce04c-023e-47d6-b5f5-22b90196e4b2") : secret "image-registry-tls" not found
May 26 11:25:05.857722 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:05.857592    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:05.857722 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:05.857661    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls podName:e73ee593-f6c2-4e31-9297-0c2c0b321c4a nodeName:}" failed. No retries permitted until 2026-05-26 11:25:09.857646545 +0000 UTC m=+42.663484997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls") pod "dns-default-5chfj" (UID: "e73ee593-f6c2-4e31-9297-0c2c0b321c4a") : secret "dns-default-metrics-tls" not found
May 26 11:25:05.859845 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:05.859823    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
May 26 11:25:05.860673 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:05.860653    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
May 26 11:25:05.860798 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:05.860766    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-68b6b\""
May 26 11:25:05.860798 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:05.860782    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
May 26 11:25:05.860922 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:05.860768    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
May 26 11:25:05.958346 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:05.958308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c716c390-14f8-4b53-9766-95deef29dbe2-signing-cabundle\") pod \"service-ca-67d667c7d-5z2cp\" (UID: \"c716c390-14f8-4b53-9766-95deef29dbe2\") " pod="openshift-service-ca/service-ca-67d667c7d-5z2cp"
May 26 11:25:05.958346 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:05.958355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-279k7\" (UniqueName: \"kubernetes.io/projected/c716c390-14f8-4b53-9766-95deef29dbe2-kube-api-access-279k7\") pod \"service-ca-67d667c7d-5z2cp\" (UID: \"c716c390-14f8-4b53-9766-95deef29dbe2\") " pod="openshift-service-ca/service-ca-67d667c7d-5z2cp"
May 26 11:25:05.958564 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:05.958442    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c716c390-14f8-4b53-9766-95deef29dbe2-signing-key\") pod \"service-ca-67d667c7d-5z2cp\" (UID: \"c716c390-14f8-4b53-9766-95deef29dbe2\") " pod="openshift-service-ca/service-ca-67d667c7d-5z2cp"
May 26 11:25:06.059802 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:06.059767    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c716c390-14f8-4b53-9766-95deef29dbe2-signing-cabundle\") pod \"service-ca-67d667c7d-5z2cp\" (UID: \"c716c390-14f8-4b53-9766-95deef29dbe2\") " pod="openshift-service-ca/service-ca-67d667c7d-5z2cp"
May 26 11:25:06.059802 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:06.059808    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-279k7\" (UniqueName: \"kubernetes.io/projected/c716c390-14f8-4b53-9766-95deef29dbe2-kube-api-access-279k7\") pod \"service-ca-67d667c7d-5z2cp\" (UID: \"c716c390-14f8-4b53-9766-95deef29dbe2\") " pod="openshift-service-ca/service-ca-67d667c7d-5z2cp"
May 26 11:25:06.060208 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:06.059840    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c716c390-14f8-4b53-9766-95deef29dbe2-signing-key\") pod \"service-ca-67d667c7d-5z2cp\" (UID: \"c716c390-14f8-4b53-9766-95deef29dbe2\") " pod="openshift-service-ca/service-ca-67d667c7d-5z2cp"
May 26 11:25:06.060443 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:06.060422    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c716c390-14f8-4b53-9766-95deef29dbe2-signing-cabundle\") pod \"service-ca-67d667c7d-5z2cp\" (UID: \"c716c390-14f8-4b53-9766-95deef29dbe2\") " pod="openshift-service-ca/service-ca-67d667c7d-5z2cp"
May 26 11:25:06.062398 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:06.062381    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c716c390-14f8-4b53-9766-95deef29dbe2-signing-key\") pod \"service-ca-67d667c7d-5z2cp\" (UID: \"c716c390-14f8-4b53-9766-95deef29dbe2\") " pod="openshift-service-ca/service-ca-67d667c7d-5z2cp"
May 26 11:25:06.067862 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:06.067837    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-279k7\" (UniqueName: \"kubernetes.io/projected/c716c390-14f8-4b53-9766-95deef29dbe2-kube-api-access-279k7\") pod \"service-ca-67d667c7d-5z2cp\" (UID: \"c716c390-14f8-4b53-9766-95deef29dbe2\") " pod="openshift-service-ca/service-ca-67d667c7d-5z2cp"
May 26 11:25:06.166661 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:06.166586    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-5z2cp"
May 26 11:25:06.242977 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:06.242945    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nqs65_9d3e68b4-5329-4f10-bf42-4f7e16a2795c/dns-node-resolver/0.log"
May 26 11:25:06.350193 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:06.349953    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-5z2cp"]
May 26 11:25:06.354080 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:25:06.354053    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc716c390_14f8_4b53_9766_95deef29dbe2.slice/crio-2690cc68cec54a5359983d0baa1df94eae6de1aa7a2e3dfcd19e06641a5e6b43 WatchSource:0}: Error finding container 2690cc68cec54a5359983d0baa1df94eae6de1aa7a2e3dfcd19e06641a5e6b43: Status 404 returned error can't find the container with id 2690cc68cec54a5359983d0baa1df94eae6de1aa7a2e3dfcd19e06641a5e6b43
May 26 11:25:06.984665 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:06.984627    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-5z2cp" event={"ID":"c716c390-14f8-4b53-9766-95deef29dbe2","Type":"ContainerStarted","Data":"2690cc68cec54a5359983d0baa1df94eae6de1aa7a2e3dfcd19e06641a5e6b43"}
May 26 11:25:07.242623 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:07.242549    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8dl7t_1061f872-9b39-430c-9fe2-e1308e621e42/node-ca/0.log"
May 26 11:25:09.891637 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:09.891592    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:09.891637 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:09.891647    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert\") pod \"ingress-canary-rtrlh\" (UID: \"3f9d7bb9-08d8-495f-9713-a6432e7f6a4f\") " pod="openshift-ingress-canary/ingress-canary-rtrlh"
May 26 11:25:09.892174 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:09.891751    2563 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:09.892174 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:09.891770    2563 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74c679c6fb-mpsq2: secret "image-registry-tls" not found
May 26 11:25:09.892174 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:09.891782    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:09.892174 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:09.891823    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls podName:195ce04c-023e-47d6-b5f5-22b90196e4b2 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:17.891806797 +0000 UTC m=+50.697645239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls") pod "image-registry-74c679c6fb-mpsq2" (UID: "195ce04c-023e-47d6-b5f5-22b90196e4b2") : secret "image-registry-tls" not found
May 26 11:25:09.892174 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:09.891867    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:09.892174 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:09.891906    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls podName:e73ee593-f6c2-4e31-9297-0c2c0b321c4a nodeName:}" failed. No retries permitted until 2026-05-26 11:25:17.891896076 +0000 UTC m=+50.697734518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls") pod "dns-default-5chfj" (UID: "e73ee593-f6c2-4e31-9297-0c2c0b321c4a") : secret "dns-default-metrics-tls" not found
May 26 11:25:09.892174 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:09.891867    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:09.892174 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:09.891933    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert podName:3f9d7bb9-08d8-495f-9713-a6432e7f6a4f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:17.891927559 +0000 UTC m=+50.697766001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert") pod "ingress-canary-rtrlh" (UID: "3f9d7bb9-08d8-495f-9713-a6432e7f6a4f") : secret "canary-serving-cert" not found
May 26 11:25:09.991335 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:09.991297    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-5z2cp" event={"ID":"c716c390-14f8-4b53-9766-95deef29dbe2","Type":"ContainerStarted","Data":"ff127b3c697315d760729fc84daeab7dfb7f225a59d5015abaf4a6ba4b2f443f"}
May 26 11:25:10.010674 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:10.010603    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-5z2cp" podStartSLOduration=2.435189769 podStartE2EDuration="5.010583872s" podCreationTimestamp="2026-05-26 11:25:05 +0000 UTC" firstStartedPulling="2026-05-26 11:25:06.356638045 +0000 UTC m=+39.162476499" lastFinishedPulling="2026-05-26 11:25:08.932032146 +0000 UTC m=+41.737870602" observedRunningTime="2026-05-26 11:25:10.009860909 +0000 UTC m=+42.815699375" watchObservedRunningTime="2026-05-26 11:25:10.010583872 +0000 UTC m=+42.816422339"
May 26 11:25:17.951253 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:17.951212    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert\") pod \"ingress-canary-rtrlh\" (UID: \"3f9d7bb9-08d8-495f-9713-a6432e7f6a4f\") " pod="openshift-ingress-canary/ingress-canary-rtrlh"
May 26 11:25:17.951712 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:17.951272    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:17.951712 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:17.951380    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:17.954492 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:17.954460    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e73ee593-f6c2-4e31-9297-0c2c0b321c4a-metrics-tls\") pod \"dns-default-5chfj\" (UID: \"e73ee593-f6c2-4e31-9297-0c2c0b321c4a\") " pod="openshift-dns/dns-default-5chfj"
May 26 11:25:17.954618 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:17.954582    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls\") pod \"image-registry-74c679c6fb-mpsq2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") " pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:17.954705 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:17.954682    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f9d7bb9-08d8-495f-9713-a6432e7f6a4f-cert\") pod \"ingress-canary-rtrlh\" (UID: \"3f9d7bb9-08d8-495f-9713-a6432e7f6a4f\") " pod="openshift-ingress-canary/ingress-canary-rtrlh"
May 26 11:25:17.963466 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:17.963444    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:17.978090 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:17.978067    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5chfj"
May 26 11:25:18.001108 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:18.001025    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rtrlh"
May 26 11:25:18.134479 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:18.134450    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74c679c6fb-mpsq2"]
May 26 11:25:18.151689 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:25:18.151663    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195ce04c_023e_47d6_b5f5_22b90196e4b2.slice/crio-e180dfe953ca6f9c77eef0e84d92a85fbd341d512b324c8fdc1cdcac5c3befe8 WatchSource:0}: Error finding container e180dfe953ca6f9c77eef0e84d92a85fbd341d512b324c8fdc1cdcac5c3befe8: Status 404 returned error can't find the container with id e180dfe953ca6f9c77eef0e84d92a85fbd341d512b324c8fdc1cdcac5c3befe8
May 26 11:25:18.161932 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:18.161889    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5chfj"]
May 26 11:25:18.166104 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:25:18.166079    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73ee593_f6c2_4e31_9297_0c2c0b321c4a.slice/crio-7b1d998f14d8d3c34154be73c48eaafc3175ab735b8260f2915ea27c6a312a80 WatchSource:0}: Error finding container 7b1d998f14d8d3c34154be73c48eaafc3175ab735b8260f2915ea27c6a312a80: Status 404 returned error can't find the container with id 7b1d998f14d8d3c34154be73c48eaafc3175ab735b8260f2915ea27c6a312a80
May 26 11:25:18.173748 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:18.173721    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rtrlh"]
May 26 11:25:18.176401 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:25:18.176381    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9d7bb9_08d8_495f_9713_a6432e7f6a4f.slice/crio-0698302cf209df1b5671b2d42548a5d8747f6549d6a3053a2ee00c25e2512c72 WatchSource:0}: Error finding container 0698302cf209df1b5671b2d42548a5d8747f6549d6a3053a2ee00c25e2512c72: Status 404 returned error can't find the container with id 0698302cf209df1b5671b2d42548a5d8747f6549d6a3053a2ee00c25e2512c72
May 26 11:25:19.011599 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:19.011550    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rtrlh" event={"ID":"3f9d7bb9-08d8-495f-9713-a6432e7f6a4f","Type":"ContainerStarted","Data":"0698302cf209df1b5671b2d42548a5d8747f6549d6a3053a2ee00c25e2512c72"}
May 26 11:25:19.013405 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:19.013355    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5chfj" event={"ID":"e73ee593-f6c2-4e31-9297-0c2c0b321c4a","Type":"ContainerStarted","Data":"7b1d998f14d8d3c34154be73c48eaafc3175ab735b8260f2915ea27c6a312a80"}
May 26 11:25:19.015329 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:19.015307    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2" event={"ID":"195ce04c-023e-47d6-b5f5-22b90196e4b2","Type":"ContainerStarted","Data":"4da899cba19066a77af721680ba06f57c2f496255972fdcb0efcd91c1948321b"}
May 26 11:25:19.015448 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:19.015335    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2" event={"ID":"195ce04c-023e-47d6-b5f5-22b90196e4b2","Type":"ContainerStarted","Data":"e180dfe953ca6f9c77eef0e84d92a85fbd341d512b324c8fdc1cdcac5c3befe8"}
May 26 11:25:19.016020 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:19.016002    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:19.038598 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:19.038552    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2" podStartSLOduration=44.038536281 podStartE2EDuration="44.038536281s" podCreationTimestamp="2026-05-26 11:24:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:25:19.0377756 +0000 UTC m=+51.843614065" watchObservedRunningTime="2026-05-26 11:25:19.038536281 +0000 UTC m=+51.844374746"
May 26 11:25:21.020567 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:21.020533    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rtrlh" event={"ID":"3f9d7bb9-08d8-495f-9713-a6432e7f6a4f","Type":"ContainerStarted","Data":"15237e3d31618de1b803125cafa330ccf14fc96dcb1f050742ea1a768fb2c297"}
May 26 11:25:21.022126 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:21.022100    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5chfj" event={"ID":"e73ee593-f6c2-4e31-9297-0c2c0b321c4a","Type":"ContainerStarted","Data":"2f34519b4029484affd214472362de3816658df465feb48da523b03d08869ff7"}
May 26 11:25:21.022231 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:21.022161    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5chfj" event={"ID":"e73ee593-f6c2-4e31-9297-0c2c0b321c4a","Type":"ContainerStarted","Data":"c2618fbd45e96f425741fc34641e2a304dac2edd7d2fbcdf55a58847860789a2"}
May 26 11:25:21.022231 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:21.022183    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5chfj"
May 26 11:25:21.035889 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:21.035848    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rtrlh" podStartSLOduration=16.800417925 podStartE2EDuration="19.035834397s" podCreationTimestamp="2026-05-26 11:25:02 +0000 UTC" firstStartedPulling="2026-05-26 11:25:18.178701363 +0000 UTC m=+50.984539806" lastFinishedPulling="2026-05-26 11:25:20.414117837 +0000 UTC m=+53.219956278" observedRunningTime="2026-05-26 11:25:21.035310817 +0000 UTC m=+53.841149284" watchObservedRunningTime="2026-05-26 11:25:21.035834397 +0000 UTC m=+53.841672860"
May 26 11:25:21.051453 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:21.051411    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-5chfj" podStartSLOduration=16.811142603 podStartE2EDuration="19.051398816s" podCreationTimestamp="2026-05-26 11:25:02 +0000 UTC" firstStartedPulling="2026-05-26 11:25:18.171229645 +0000 UTC m=+50.977068087" lastFinishedPulling="2026-05-26 11:25:20.411485647 +0000 UTC m=+53.217324300" observedRunningTime="2026-05-26 11:25:21.05104665 +0000 UTC m=+53.856885113" watchObservedRunningTime="2026-05-26 11:25:21.051398816 +0000 UTC m=+53.857237309"
May 26 11:25:26.968299 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:26.968245    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dkfdl"
May 26 11:25:27.455598 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.455568    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74c679c6fb-mpsq2"]
May 26 11:25:27.527324 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.527272    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rhx5w"]
May 26 11:25:27.530734 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.530714    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.534027 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.534007    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 26 11:25:27.534140 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.534064    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 26 11:25:27.534272 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.534168    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2mb7j\""
May 26 11:25:27.534772 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.534758    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 26 11:25:27.534772 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.534765    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 26 11:25:27.544088 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.544069    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rhx5w"]
May 26 11:25:27.626133 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.626100    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fba481c9-7892-4902-aa77-9eb92d9e38b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhx5w\" (UID: \"fba481c9-7892-4902-aa77-9eb92d9e38b6\") " pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.626317 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.626164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvvwj\" (UniqueName: \"kubernetes.io/projected/fba481c9-7892-4902-aa77-9eb92d9e38b6-kube-api-access-xvvwj\") pod \"insights-runtime-extractor-rhx5w\" (UID: \"fba481c9-7892-4902-aa77-9eb92d9e38b6\") " pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.626317 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.626210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fba481c9-7892-4902-aa77-9eb92d9e38b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhx5w\" (UID: \"fba481c9-7892-4902-aa77-9eb92d9e38b6\") " pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.626317 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.626241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fba481c9-7892-4902-aa77-9eb92d9e38b6-data-volume\") pod \"insights-runtime-extractor-rhx5w\" (UID: \"fba481c9-7892-4902-aa77-9eb92d9e38b6\") " pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.626453 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.626324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fba481c9-7892-4902-aa77-9eb92d9e38b6-crio-socket\") pod \"insights-runtime-extractor-rhx5w\" (UID: \"fba481c9-7892-4902-aa77-9eb92d9e38b6\") " pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.726651 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.726580    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fba481c9-7892-4902-aa77-9eb92d9e38b6-data-volume\") pod \"insights-runtime-extractor-rhx5w\" (UID: \"fba481c9-7892-4902-aa77-9eb92d9e38b6\") " pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.726651 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.726630    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fba481c9-7892-4902-aa77-9eb92d9e38b6-crio-socket\") pod \"insights-runtime-extractor-rhx5w\" (UID: \"fba481c9-7892-4902-aa77-9eb92d9e38b6\") " pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.726856 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.726675    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fba481c9-7892-4902-aa77-9eb92d9e38b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhx5w\" (UID: \"fba481c9-7892-4902-aa77-9eb92d9e38b6\") " pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.726856 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.726710    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xvvwj\" (UniqueName: \"kubernetes.io/projected/fba481c9-7892-4902-aa77-9eb92d9e38b6-kube-api-access-xvvwj\") pod \"insights-runtime-extractor-rhx5w\" (UID: \"fba481c9-7892-4902-aa77-9eb92d9e38b6\") " pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.726856 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.726842    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fba481c9-7892-4902-aa77-9eb92d9e38b6-crio-socket\") pod \"insights-runtime-extractor-rhx5w\" (UID: \"fba481c9-7892-4902-aa77-9eb92d9e38b6\") " pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.726994 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.726862    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fba481c9-7892-4902-aa77-9eb92d9e38b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhx5w\" (UID: \"fba481c9-7892-4902-aa77-9eb92d9e38b6\") " pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.726994 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.726926    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fba481c9-7892-4902-aa77-9eb92d9e38b6-data-volume\") pod \"insights-runtime-extractor-rhx5w\" (UID: \"fba481c9-7892-4902-aa77-9eb92d9e38b6\") " pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.729463 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.729442    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 26 11:25:27.729905 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.729890    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 26 11:25:27.737446 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.737425    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fba481c9-7892-4902-aa77-9eb92d9e38b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhx5w\" (UID: \"fba481c9-7892-4902-aa77-9eb92d9e38b6\") " pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.739071 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.739050    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fba481c9-7892-4902-aa77-9eb92d9e38b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhx5w\" (UID: \"fba481c9-7892-4902-aa77-9eb92d9e38b6\") " pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.744117 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.744099    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 26 11:25:27.754895 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.754880    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 26 11:25:27.764947 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.764927    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvvwj\" (UniqueName: \"kubernetes.io/projected/fba481c9-7892-4902-aa77-9eb92d9e38b6-kube-api-access-xvvwj\") pod \"insights-runtime-extractor-rhx5w\" (UID: \"fba481c9-7892-4902-aa77-9eb92d9e38b6\") " pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.841392 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.841364    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2mb7j\""
May 26 11:25:27.849273 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.849254    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rhx5w"
May 26 11:25:27.989967 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:27.989940    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rhx5w"]
May 26 11:25:27.993456 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:25:27.993421    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba481c9_7892_4902_aa77_9eb92d9e38b6.slice/crio-2efbe95530c67db6f6af0eadaaf8464312d5c03e2967d2ac8b9b89363ab14bec WatchSource:0}: Error finding container 2efbe95530c67db6f6af0eadaaf8464312d5c03e2967d2ac8b9b89363ab14bec: Status 404 returned error can't find the container with id 2efbe95530c67db6f6af0eadaaf8464312d5c03e2967d2ac8b9b89363ab14bec
May 26 11:25:28.037707 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:28.037682    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rhx5w" event={"ID":"fba481c9-7892-4902-aa77-9eb92d9e38b6","Type":"ContainerStarted","Data":"2efbe95530c67db6f6af0eadaaf8464312d5c03e2967d2ac8b9b89363ab14bec"}
May 26 11:25:28.704436 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:28.704394    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p4cml"]
May 26 11:25:28.706102 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:28.706082    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p4cml"
May 26 11:25:28.709474 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:28.709451    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 26 11:25:28.710189 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:28.710169    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-24zwb\""
May 26 11:25:28.717905 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:28.717880    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p4cml"]
May 26 11:25:28.734636 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:28.734609    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/df896a28-f4ca-4dd4-bd50-39894600be5d-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-p4cml\" (UID: \"df896a28-f4ca-4dd4-bd50-39894600be5d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p4cml"
May 26 11:25:28.835816 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:28.835780    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/df896a28-f4ca-4dd4-bd50-39894600be5d-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-p4cml\" (UID: \"df896a28-f4ca-4dd4-bd50-39894600be5d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p4cml"
May 26 11:25:28.835959 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:28.835906    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
May 26 11:25:28.836010 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:28.835961    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df896a28-f4ca-4dd4-bd50-39894600be5d-tls-certificates podName:df896a28-f4ca-4dd4-bd50-39894600be5d nodeName:}" failed. No retries permitted until 2026-05-26 11:25:29.335947398 +0000 UTC m=+62.141785839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/df896a28-f4ca-4dd4-bd50-39894600be5d-tls-certificates") pod "prometheus-operator-admission-webhook-6c4bbc67b7-p4cml" (UID: "df896a28-f4ca-4dd4-bd50-39894600be5d") : secret "prometheus-operator-admission-webhook-tls" not found
May 26 11:25:29.044390 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:29.044355    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rhx5w" event={"ID":"fba481c9-7892-4902-aa77-9eb92d9e38b6","Type":"ContainerStarted","Data":"f5c6bc426af5caee0df3e585da7968f4696eca2460795ca1d6e0da6c12c2ed1c"}
May 26 11:25:29.044390 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:29.044391    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rhx5w" event={"ID":"fba481c9-7892-4902-aa77-9eb92d9e38b6","Type":"ContainerStarted","Data":"7ef27c76f31a7b42793957164579ca49514ba7ac7a7b43c46342814fe4a7432b"}
May 26 11:25:29.339507 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:29.339409    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/df896a28-f4ca-4dd4-bd50-39894600be5d-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-p4cml\" (UID: \"df896a28-f4ca-4dd4-bd50-39894600be5d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p4cml"
May 26 11:25:29.342262 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:29.342229    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/df896a28-f4ca-4dd4-bd50-39894600be5d-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-p4cml\" (UID: \"df896a28-f4ca-4dd4-bd50-39894600be5d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p4cml"
May 26 11:25:29.614883 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:29.614806    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p4cml"
May 26 11:25:29.745866 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:29.745838    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p4cml"]
May 26 11:25:29.749024 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:25:29.748996    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf896a28_f4ca_4dd4_bd50_39894600be5d.slice/crio-f6d6366c1d76fda73624b7e0b797f86f4838f1ea4ae104ed4e0f6c18a2e3f686 WatchSource:0}: Error finding container f6d6366c1d76fda73624b7e0b797f86f4838f1ea4ae104ed4e0f6c18a2e3f686: Status 404 returned error can't find the container with id f6d6366c1d76fda73624b7e0b797f86f4838f1ea4ae104ed4e0f6c18a2e3f686
May 26 11:25:30.052412 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:30.052368    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p4cml" event={"ID":"df896a28-f4ca-4dd4-bd50-39894600be5d","Type":"ContainerStarted","Data":"f6d6366c1d76fda73624b7e0b797f86f4838f1ea4ae104ed4e0f6c18a2e3f686"}
May 26 11:25:31.026453 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:31.026422    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5chfj"
May 26 11:25:31.058511 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:31.058476    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rhx5w" event={"ID":"fba481c9-7892-4902-aa77-9eb92d9e38b6","Type":"ContainerStarted","Data":"aff78b80da71a4cf09a8c5817059b45d890f0b74e6d8fae76516daf738eed20e"}
May 26 11:25:31.077297 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:31.077133    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rhx5w" podStartSLOduration=1.87670089 podStartE2EDuration="4.077115909s" podCreationTimestamp="2026-05-26 11:25:27 +0000 UTC" firstStartedPulling="2026-05-26 11:25:28.043754323 +0000 UTC m=+60.849592770" lastFinishedPulling="2026-05-26 11:25:30.244169347 +0000 UTC m=+63.050007789" observedRunningTime="2026-05-26 11:25:31.076974068 +0000 UTC m=+63.882812537" watchObservedRunningTime="2026-05-26 11:25:31.077115909 +0000 UTC m=+63.882954374"
May 26 11:25:32.062542 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.062509    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p4cml" event={"ID":"df896a28-f4ca-4dd4-bd50-39894600be5d","Type":"ContainerStarted","Data":"ac21e76d16373e5a6bc5359d410014e9926dea6af8097d9d303086638093d82f"}
May 26 11:25:32.062933 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.062868    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p4cml"
May 26 11:25:32.068469 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.068445    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p4cml"
May 26 11:25:32.093146 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.093107    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p4cml" podStartSLOduration=2.7678275279999998 podStartE2EDuration="4.093095846s" podCreationTimestamp="2026-05-26 11:25:28 +0000 UTC" firstStartedPulling="2026-05-26 11:25:29.751002378 +0000 UTC m=+62.556840824" lastFinishedPulling="2026-05-26 11:25:31.076270683 +0000 UTC m=+63.882109142" observedRunningTime="2026-05-26 11:25:32.077818653 +0000 UTC m=+64.883657119" watchObservedRunningTime="2026-05-26 11:25:32.093095846 +0000 UTC m=+64.898934310"
May 26 11:25:32.465051 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.464958    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs\") pod \"network-metrics-daemon-vgwsb\" (UID: \"a9201d70-b657-4adf-9d01-5363b9fb7ded\") " pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:25:32.467823 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.467802    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 11:25:32.477850 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.477828    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9201d70-b657-4adf-9d01-5363b9fb7ded-metrics-certs\") pod \"network-metrics-daemon-vgwsb\" (UID: \"a9201d70-b657-4adf-9d01-5363b9fb7ded\") " pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:25:32.504459 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.504432    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xvpvn\""
May 26 11:25:32.512625 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.512610    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgwsb"
May 26 11:25:32.565705 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.565377    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-52sp9\" (UniqueName: \"kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9\") pod \"network-check-target-5pwxn\" (UID: \"44da6752-3b21-42e6-942c-ab7fc44713dc\") " pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:25:32.565705 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.565453    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret\") pod \"global-pull-secret-syncer-wzshh\" (UID: \"7f39775e-7a53-44fb-8e18-70fd6a5764c5\") " pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:25:32.569478 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.569454    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 11:25:32.569478 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.569472    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 11:25:32.578068 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.578046    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f39775e-7a53-44fb-8e18-70fd6a5764c5-original-pull-secret\") pod \"global-pull-secret-syncer-wzshh\" (UID: \"7f39775e-7a53-44fb-8e18-70fd6a5764c5\") " pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:25:32.578787 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.578770    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 11:25:32.589490 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.589464    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52sp9\" (UniqueName: \"kubernetes.io/projected/44da6752-3b21-42e6-942c-ab7fc44713dc-kube-api-access-52sp9\") pod \"network-check-target-5pwxn\" (UID: \"44da6752-3b21-42e6-942c-ab7fc44713dc\") " pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:25:32.636909 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.636877    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vgwsb"]
May 26 11:25:32.640091 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:25:32.640065    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9201d70_b657_4adf_9d01_5363b9fb7ded.slice/crio-39dba5d09b5cd39c535c147e3a848bcf266f39e8a060a4e39c574c3cdcfa250e WatchSource:0}: Error finding container 39dba5d09b5cd39c535c147e3a848bcf266f39e8a060a4e39c574c3cdcfa250e: Status 404 returned error can't find the container with id 39dba5d09b5cd39c535c147e3a848bcf266f39e8a060a4e39c574c3cdcfa250e
May 26 11:25:32.809672 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.809639    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zzwv5\""
May 26 11:25:32.811465 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.811452    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzshh"
May 26 11:25:32.818086 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.818048    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:25:32.945837 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.945799    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wzshh"]
May 26 11:25:32.949273 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:25:32.949241    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f39775e_7a53_44fb_8e18_70fd6a5764c5.slice/crio-e093b19148ecd8925fd812522d35732b7c6c36b40103097bd883aeb3008864ac WatchSource:0}: Error finding container e093b19148ecd8925fd812522d35732b7c6c36b40103097bd883aeb3008864ac: Status 404 returned error can't find the container with id e093b19148ecd8925fd812522d35732b7c6c36b40103097bd883aeb3008864ac
May 26 11:25:32.962426 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:32.962401    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5pwxn"]
May 26 11:25:32.966267 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:25:32.966243    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44da6752_3b21_42e6_942c_ab7fc44713dc.slice/crio-48225ba3b675e0c20c57890e85022b398e8979c3abbdc2cc07c0c34be644034e WatchSource:0}: Error finding container 48225ba3b675e0c20c57890e85022b398e8979c3abbdc2cc07c0c34be644034e: Status 404 returned error can't find the container with id 48225ba3b675e0c20c57890e85022b398e8979c3abbdc2cc07c0c34be644034e
May 26 11:25:33.066379 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:33.066310    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5pwxn" event={"ID":"44da6752-3b21-42e6-942c-ab7fc44713dc","Type":"ContainerStarted","Data":"48225ba3b675e0c20c57890e85022b398e8979c3abbdc2cc07c0c34be644034e"}
May 26 11:25:33.067396 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:33.067374    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wzshh" event={"ID":"7f39775e-7a53-44fb-8e18-70fd6a5764c5","Type":"ContainerStarted","Data":"e093b19148ecd8925fd812522d35732b7c6c36b40103097bd883aeb3008864ac"}
May 26 11:25:33.068443 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:33.068420    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vgwsb" event={"ID":"a9201d70-b657-4adf-9d01-5363b9fb7ded","Type":"ContainerStarted","Data":"39dba5d09b5cd39c535c147e3a848bcf266f39e8a060a4e39c574c3cdcfa250e"}
May 26 11:25:35.079052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:35.079013    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vgwsb" event={"ID":"a9201d70-b657-4adf-9d01-5363b9fb7ded","Type":"ContainerStarted","Data":"535006fca0ff2590315cc6ca13699d30ceb1e854181f8c40ab80e170c0b1f1fc"}
May 26 11:25:35.079052 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:35.079061    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vgwsb" event={"ID":"a9201d70-b657-4adf-9d01-5363b9fb7ded","Type":"ContainerStarted","Data":"ee08e9088393d581e1f6368364f91b580310a4fc38e69719c763611b68b1dfc8"}
May 26 11:25:35.097249 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:35.097134    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vgwsb" podStartSLOduration=66.563183091 podStartE2EDuration="1m8.097115242s" podCreationTimestamp="2026-05-26 11:24:27 +0000 UTC" firstStartedPulling="2026-05-26 11:25:32.641895792 +0000 UTC m=+65.447734234" lastFinishedPulling="2026-05-26 11:25:34.175827925 +0000 UTC m=+66.981666385" observedRunningTime="2026-05-26 11:25:35.095203106 +0000 UTC m=+67.901041569" watchObservedRunningTime="2026-05-26 11:25:35.097115242 +0000 UTC m=+67.902953704"
May 26 11:25:37.461998 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:37.461955    2563 patch_prober.go:28] interesting pod/image-registry-74c679c6fb-mpsq2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 26 11:25:37.462600 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:37.462025    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2" podUID="195ce04c-023e-47d6-b5f5-22b90196e4b2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:25:38.088374 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:38.088335    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5pwxn" event={"ID":"44da6752-3b21-42e6-942c-ab7fc44713dc","Type":"ContainerStarted","Data":"771eb81beb59295695f63e626cf21a7c486b666a8ef0bd36d8da9271bbf92fc2"}
May 26 11:25:38.088543 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:38.088427    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:25:38.091510 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:38.091475    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wzshh" event={"ID":"7f39775e-7a53-44fb-8e18-70fd6a5764c5","Type":"ContainerStarted","Data":"3412fd45adc3627d2813fb0fc82acf18f2e7fdbc46ac526ddcff1dae0eebc94e"}
May 26 11:25:38.104212 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:38.104166    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5pwxn" podStartSLOduration=66.517865087 podStartE2EDuration="1m11.104154635s" podCreationTimestamp="2026-05-26 11:24:27 +0000 UTC" firstStartedPulling="2026-05-26 11:25:32.968120264 +0000 UTC m=+65.773958706" lastFinishedPulling="2026-05-26 11:25:37.554409803 +0000 UTC m=+70.360248254" observedRunningTime="2026-05-26 11:25:38.103407545 +0000 UTC m=+70.909246009" watchObservedRunningTime="2026-05-26 11:25:38.104154635 +0000 UTC m=+70.909993119"
May 26 11:25:38.117754 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:38.117710    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wzshh" podStartSLOduration=65.511770845 podStartE2EDuration="1m10.117698767s" podCreationTimestamp="2026-05-26 11:24:28 +0000 UTC" firstStartedPulling="2026-05-26 11:25:32.951188658 +0000 UTC m=+65.757027103" lastFinishedPulling="2026-05-26 11:25:37.557116569 +0000 UTC m=+70.362955025" observedRunningTime="2026-05-26 11:25:38.117460505 +0000 UTC m=+70.923298971" watchObservedRunningTime="2026-05-26 11:25:38.117698767 +0000 UTC m=+70.923537232"
May 26 11:25:39.124973 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.124938    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"]
May 26 11:25:39.128123 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.128100    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"
May 26 11:25:39.130769 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.130746    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 26 11:25:39.131699 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.131677    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 26 11:25:39.131793 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.131703    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 26 11:25:39.131793 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.131703    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 26 11:25:39.131912 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.131824    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 26 11:25:39.131999 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.131984    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nfwzp\""
May 26 11:25:39.142199 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.142179    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"]
May 26 11:25:39.170514 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.170487    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-949ct"]
May 26 11:25:39.173832 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.173813    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"]
May 26 11:25:39.174005 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.173987    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.176175 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.176154    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 26 11:25:39.176456 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.176435    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 26 11:25:39.176545 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.176435    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 26 11:25:39.176619 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.176474    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-klwwf\""
May 26 11:25:39.177065 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.177043    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.179369 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.179346    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 26 11:25:39.179498 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.179424    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 26 11:25:39.179498 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.179354    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vxq2c\""
May 26 11:25:39.179625 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.179564    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 26 11:25:39.187183 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.186928    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"]
May 26 11:25:39.219881 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.219854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.220003 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.219887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfxp5\" (UniqueName: \"kubernetes.io/projected/93f480d3-f56c-4fa5-9ad4-dd2252a70d5a-kube-api-access-dfxp5\") pod \"openshift-state-metrics-9c959559f-dbfkl\" (UID: \"93f480d3-f56c-4fa5-9ad4-dd2252a70d5a\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"
May 26 11:25:39.220003 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.219936    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-tls\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.220003 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.219967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/93f480d3-f56c-4fa5-9ad4-dd2252a70d5a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-dbfkl\" (UID: \"93f480d3-f56c-4fa5-9ad4-dd2252a70d5a\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"
May 26 11:25:39.220164 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.220012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/37858259-5496-44a7-984d-78eafc49f221-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.220164 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.220084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37858259-5496-44a7-984d-78eafc49f221-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.220272 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.220207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/37858259-5496-44a7-984d-78eafc49f221-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.220272 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.220261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/486de637-3719-41d0-bc8a-d1969384daa3-root\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.220374 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.220309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93f480d3-f56c-4fa5-9ad4-dd2252a70d5a-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-dbfkl\" (UID: \"93f480d3-f56c-4fa5-9ad4-dd2252a70d5a\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"
May 26 11:25:39.220374 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.220351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37858259-5496-44a7-984d-78eafc49f221-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.220450 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.220374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxf6m\" (UniqueName: \"kubernetes.io/projected/37858259-5496-44a7-984d-78eafc49f221-kube-api-access-kxf6m\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.220450 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.220390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/486de637-3719-41d0-bc8a-d1969384daa3-sys\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.220450 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.220415    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-wtmp\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.220450 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.220446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/486de637-3719-41d0-bc8a-d1969384daa3-metrics-client-ca\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.220596 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.220473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93f480d3-f56c-4fa5-9ad4-dd2252a70d5a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-dbfkl\" (UID: \"93f480d3-f56c-4fa5-9ad4-dd2252a70d5a\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"
May 26 11:25:39.220596 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.220504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-textfile\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.220596 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.220529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk7ts\" (UniqueName: \"kubernetes.io/projected/486de637-3719-41d0-bc8a-d1969384daa3-kube-api-access-zk7ts\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.220596 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.220575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/37858259-5496-44a7-984d-78eafc49f221-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.220728 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.220602    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-accelerators-collector-config\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.321859 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.321822    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/93f480d3-f56c-4fa5-9ad4-dd2252a70d5a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-dbfkl\" (UID: \"93f480d3-f56c-4fa5-9ad4-dd2252a70d5a\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"
May 26 11:25:39.322034 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.321874    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/37858259-5496-44a7-984d-78eafc49f221-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.322034 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.321904    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37858259-5496-44a7-984d-78eafc49f221-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.322034 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.321937    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/37858259-5496-44a7-984d-78eafc49f221-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.322034 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.321967    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/486de637-3719-41d0-bc8a-d1969384daa3-root\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.322034 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.321999    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93f480d3-f56c-4fa5-9ad4-dd2252a70d5a-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-dbfkl\" (UID: \"93f480d3-f56c-4fa5-9ad4-dd2252a70d5a\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"
May 26 11:25:39.322328 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322042    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37858259-5496-44a7-984d-78eafc49f221-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.322328 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322085    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kxf6m\" (UniqueName: \"kubernetes.io/projected/37858259-5496-44a7-984d-78eafc49f221-kube-api-access-kxf6m\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.322328 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322113    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/486de637-3719-41d0-bc8a-d1969384daa3-sys\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.322328 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322139    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-wtmp\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.322328 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322164    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/486de637-3719-41d0-bc8a-d1969384daa3-metrics-client-ca\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.322328 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322193    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93f480d3-f56c-4fa5-9ad4-dd2252a70d5a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-dbfkl\" (UID: \"93f480d3-f56c-4fa5-9ad4-dd2252a70d5a\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"
May 26 11:25:39.322328 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322227    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-textfile\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.322328 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322260    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zk7ts\" (UniqueName: \"kubernetes.io/projected/486de637-3719-41d0-bc8a-d1969384daa3-kube-api-access-zk7ts\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.322328 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322274    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/37858259-5496-44a7-984d-78eafc49f221-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.322748 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322312    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/37858259-5496-44a7-984d-78eafc49f221-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.322748 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322379    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-accelerators-collector-config\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.322748 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:39.322395    2563 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
May 26 11:25:39.322748 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322419    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.322748 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:39.322454    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37858259-5496-44a7-984d-78eafc49f221-kube-state-metrics-tls podName:37858259-5496-44a7-984d-78eafc49f221 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:39.822434107 +0000 UTC m=+72.628272555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/37858259-5496-44a7-984d-78eafc49f221-kube-state-metrics-tls") pod "kube-state-metrics-54fd6bc5c4-277l9" (UID: "37858259-5496-44a7-984d-78eafc49f221") : secret "kube-state-metrics-tls" not found
May 26 11:25:39.322748 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322474    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxp5\" (UniqueName: \"kubernetes.io/projected/93f480d3-f56c-4fa5-9ad4-dd2252a70d5a-kube-api-access-dfxp5\") pod \"openshift-state-metrics-9c959559f-dbfkl\" (UID: \"93f480d3-f56c-4fa5-9ad4-dd2252a70d5a\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"
May 26 11:25:39.322748 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322515    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-tls\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.322748 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:39.322614    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 26 11:25:39.322748 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:39.322647    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-tls podName:486de637-3719-41d0-bc8a-d1969384daa3 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:39.822636372 +0000 UTC m=+72.628474820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-tls") pod "node-exporter-949ct" (UID: "486de637-3719-41d0-bc8a-d1969384daa3") : secret "node-exporter-tls" not found
May 26 11:25:39.322748 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322682    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37858259-5496-44a7-984d-78eafc49f221-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.322748 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322692    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/486de637-3719-41d0-bc8a-d1969384daa3-sys\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.323297 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322813    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-wtmp\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.323297 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.322862    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/486de637-3719-41d0-bc8a-d1969384daa3-root\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.323297 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.323160    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-accelerators-collector-config\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.323472 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.323443    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/37858259-5496-44a7-984d-78eafc49f221-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.323717 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.323695    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-textfile\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.323984 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.323959    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/486de637-3719-41d0-bc8a-d1969384daa3-metrics-client-ca\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.324215 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.324194    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93f480d3-f56c-4fa5-9ad4-dd2252a70d5a-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-dbfkl\" (UID: \"93f480d3-f56c-4fa5-9ad4-dd2252a70d5a\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"
May 26 11:25:39.324982 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.324940    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/93f480d3-f56c-4fa5-9ad4-dd2252a70d5a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-dbfkl\" (UID: \"93f480d3-f56c-4fa5-9ad4-dd2252a70d5a\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"
May 26 11:25:39.325222 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.325198    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.325782 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.325757    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37858259-5496-44a7-984d-78eafc49f221-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.326263 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.326242    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93f480d3-f56c-4fa5-9ad4-dd2252a70d5a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-dbfkl\" (UID: \"93f480d3-f56c-4fa5-9ad4-dd2252a70d5a\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"
May 26 11:25:39.333698 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.333664    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk7ts\" (UniqueName: \"kubernetes.io/projected/486de637-3719-41d0-bc8a-d1969384daa3-kube-api-access-zk7ts\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.335172 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.335153    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxf6m\" (UniqueName: \"kubernetes.io/projected/37858259-5496-44a7-984d-78eafc49f221-kube-api-access-kxf6m\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.335265 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.335207    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxp5\" (UniqueName: \"kubernetes.io/projected/93f480d3-f56c-4fa5-9ad4-dd2252a70d5a-kube-api-access-dfxp5\") pod \"openshift-state-metrics-9c959559f-dbfkl\" (UID: \"93f480d3-f56c-4fa5-9ad4-dd2252a70d5a\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"
May 26 11:25:39.438343 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.438248    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"
May 26 11:25:39.565915 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.565748    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl"]
May 26 11:25:39.570700 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:25:39.570665    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f480d3_f56c_4fa5_9ad4_dd2252a70d5a.slice/crio-57f1c800134f050675dffa520c1c0141ed6a2a0e8d6905a855efab6be5c2f993 WatchSource:0}: Error finding container 57f1c800134f050675dffa520c1c0141ed6a2a0e8d6905a855efab6be5c2f993: Status 404 returned error can't find the container with id 57f1c800134f050675dffa520c1c0141ed6a2a0e8d6905a855efab6be5c2f993
May 26 11:25:39.762334 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.762278    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-59cb6d89c7-qrjpj"]
May 26 11:25:39.765899 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.765882    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.768268 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.768241    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 26 11:25:39.768533 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.768513    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 26 11:25:39.768640 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.768593    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 26 11:25:39.768689 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.768639    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 26 11:25:39.768778 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.768758    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 26 11:25:39.768778 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.768773    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 26 11:25:39.768901 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.768770    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lbmdv\""
May 26 11:25:39.768901 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.768858    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 26 11:25:39.773557 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.773536    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59cb6d89c7-qrjpj"]
May 26 11:25:39.826491 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.826459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-console-config\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.826491 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.826495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/516e577e-045a-476d-a62d-16dc0c999a91-console-oauth-config\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.826695 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.826513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-oauth-serving-cert\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.826695 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.826538    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/37858259-5496-44a7-984d-78eafc49f221-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.826695 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.826558    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-service-ca\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.826695 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.826611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/516e577e-045a-476d-a62d-16dc0c999a91-console-serving-cert\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.826695 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.826658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc2b5\" (UniqueName: \"kubernetes.io/projected/516e577e-045a-476d-a62d-16dc0c999a91-kube-api-access-rc2b5\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.826894 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.826718    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-tls\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.829038 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.828991    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/486de637-3719-41d0-bc8a-d1969384daa3-node-exporter-tls\") pod \"node-exporter-949ct\" (UID: \"486de637-3719-41d0-bc8a-d1969384daa3\") " pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:39.829134 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.829074    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/37858259-5496-44a7-984d-78eafc49f221-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-277l9\" (UID: \"37858259-5496-44a7-984d-78eafc49f221\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:39.927755 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.927630    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-console-config\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.927755 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.927665    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/516e577e-045a-476d-a62d-16dc0c999a91-console-oauth-config\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.927755 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.927682    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-oauth-serving-cert\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.927755 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.927700    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-service-ca\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.927755 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.927716    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/516e577e-045a-476d-a62d-16dc0c999a91-console-serving-cert\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.928057 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.928011    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rc2b5\" (UniqueName: \"kubernetes.io/projected/516e577e-045a-476d-a62d-16dc0c999a91-kube-api-access-rc2b5\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.928465 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.928442    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-console-config\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.928465 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.928461    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-oauth-serving-cert\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.928613 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.928473    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-service-ca\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.930145 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.930125    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/516e577e-045a-476d-a62d-16dc0c999a91-console-oauth-config\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.930208 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.930181    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/516e577e-045a-476d-a62d-16dc0c999a91-console-serving-cert\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:39.936747 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:39.936727    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc2b5\" (UniqueName: \"kubernetes.io/projected/516e577e-045a-476d-a62d-16dc0c999a91-kube-api-access-rc2b5\") pod \"console-59cb6d89c7-qrjpj\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") " pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:40.075644 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.075607    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:40.084517 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.084494    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-949ct"
May 26 11:25:40.091308 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.091232    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"
May 26 11:25:40.096334 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:25:40.095936    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486de637_3719_41d0_bc8a_d1969384daa3.slice/crio-2e9ec438dd602007d3b91102245773bb341fef95667c227e51ce249ad9d8698f WatchSource:0}: Error finding container 2e9ec438dd602007d3b91102245773bb341fef95667c227e51ce249ad9d8698f: Status 404 returned error can't find the container with id 2e9ec438dd602007d3b91102245773bb341fef95667c227e51ce249ad9d8698f
May 26 11:25:40.101234 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.101201    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl" event={"ID":"93f480d3-f56c-4fa5-9ad4-dd2252a70d5a","Type":"ContainerStarted","Data":"4c884bb4d3b98aa97d49b7897d92752408b03e4db937c6ec79f3c7639bf200c5"}
May 26 11:25:40.101370 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.101245    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl" event={"ID":"93f480d3-f56c-4fa5-9ad4-dd2252a70d5a","Type":"ContainerStarted","Data":"eb919b9e7a93baa9a6be628ff715ae05e02929895dee21580a849c1e6646c648"}
May 26 11:25:40.101370 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.101259    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl" event={"ID":"93f480d3-f56c-4fa5-9ad4-dd2252a70d5a","Type":"ContainerStarted","Data":"57f1c800134f050675dffa520c1c0141ed6a2a0e8d6905a855efab6be5c2f993"}
May 26 11:25:40.241535 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.241407    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59cb6d89c7-qrjpj"]
May 26 11:25:40.244134 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:25:40.244102    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod516e577e_045a_476d_a62d_16dc0c999a91.slice/crio-9ea51cdaa85bca9024ebc6e5747f94ba065f873e90aeed2c8e51fb8e858a8e2b WatchSource:0}: Error finding container 9ea51cdaa85bca9024ebc6e5747f94ba065f873e90aeed2c8e51fb8e858a8e2b: Status 404 returned error can't find the container with id 9ea51cdaa85bca9024ebc6e5747f94ba065f873e90aeed2c8e51fb8e858a8e2b
May 26 11:25:40.260792 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.260769    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:25:40.267331 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.267310    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9"]
May 26 11:25:40.267420 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:25:40.267398    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37858259_5496_44a7_984d_78eafc49f221.slice/crio-c58834d3966519a464e903625e58f705390e62737133376bff74dc582888f989 WatchSource:0}: Error finding container c58834d3966519a464e903625e58f705390e62737133376bff74dc582888f989: Status 404 returned error can't find the container with id c58834d3966519a464e903625e58f705390e62737133376bff74dc582888f989
May 26 11:25:40.267468 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.267442    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.269871 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.269853    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 26 11:25:40.269957 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.269877    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hg54s\""
May 26 11:25:40.269957 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.269906    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 26 11:25:40.269957 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.269856    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 26 11:25:40.270134 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.270104    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 26 11:25:40.270260 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.270183    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 26 11:25:40.270365 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.270342    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 26 11:25:40.270485 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.270469    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 26 11:25:40.270705 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.270687    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 26 11:25:40.270781 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.270762    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 26 11:25:40.277166 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.277139    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:25:40.332238 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.332167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.332413 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.332244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.332413 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.332299    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b8dq\" (UniqueName: \"kubernetes.io/projected/990dfe56-b4c6-460d-8883-dd857554e019-kube-api-access-4b8dq\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.332413 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.332362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/990dfe56-b4c6-460d-8883-dd857554e019-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.332413 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.332409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/990dfe56-b4c6-460d-8883-dd857554e019-config-out\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.332625 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.332437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.332625 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.332461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-web-config\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.332625 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.332502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/990dfe56-b4c6-460d-8883-dd857554e019-tls-assets\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.332625 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.332551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.332625 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.332582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/990dfe56-b4c6-460d-8883-dd857554e019-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.332866 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.332755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-config-volume\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.332866 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.332799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/990dfe56-b4c6-460d-8883-dd857554e019-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.332866 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.332842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.434234 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.434190    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.434234 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.434236    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/990dfe56-b4c6-460d-8883-dd857554e019-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.434472 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.434262    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-config-volume\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.434472 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.434305    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/990dfe56-b4c6-460d-8883-dd857554e019-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.434472 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.434332    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.434472 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.434372    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.434472 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.434397    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.434472 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.434421    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4b8dq\" (UniqueName: \"kubernetes.io/projected/990dfe56-b4c6-460d-8883-dd857554e019-kube-api-access-4b8dq\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.434472 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.434464    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/990dfe56-b4c6-460d-8883-dd857554e019-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.434797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.434505    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/990dfe56-b4c6-460d-8883-dd857554e019-config-out\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.434797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.434532    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.434797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.434553    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-web-config\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.434797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.434601    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/990dfe56-b4c6-460d-8883-dd857554e019-tls-assets\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.434797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.434653    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/990dfe56-b4c6-460d-8883-dd857554e019-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.436268 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.436237    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/990dfe56-b4c6-460d-8883-dd857554e019-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.436405 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.436242    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/990dfe56-b4c6-460d-8883-dd857554e019-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.438027 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.437999    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/990dfe56-b4c6-460d-8883-dd857554e019-config-out\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.438350 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.438249    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.438476 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.438410    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.438476 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.438444    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/990dfe56-b4c6-460d-8883-dd857554e019-tls-assets\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.438627 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.438600    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-config-volume\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.438786 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.438757    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.439817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.439795    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-web-config\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.439898 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.439846    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.440068 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.440038    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.442342 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.442316    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b8dq\" (UniqueName: \"kubernetes.io/projected/990dfe56-b4c6-460d-8883-dd857554e019-kube-api-access-4b8dq\") pod \"alertmanager-main-0\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:40.577076 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:40.577039    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 11:25:41.109201 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.109150    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-949ct" event={"ID":"486de637-3719-41d0-bc8a-d1969384daa3","Type":"ContainerStarted","Data":"2e9ec438dd602007d3b91102245773bb341fef95667c227e51ce249ad9d8698f"}
May 26 11:25:41.112765 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.112705    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59cb6d89c7-qrjpj" event={"ID":"516e577e-045a-476d-a62d-16dc0c999a91","Type":"ContainerStarted","Data":"9ea51cdaa85bca9024ebc6e5747f94ba065f873e90aeed2c8e51fb8e858a8e2b"}
May 26 11:25:41.114176 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.114130    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9" event={"ID":"37858259-5496-44a7-984d-78eafc49f221","Type":"ContainerStarted","Data":"c58834d3966519a464e903625e58f705390e62737133376bff74dc582888f989"}
May 26 11:25:41.210121 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.210062    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:25:41.212893 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:25:41.212835    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990dfe56_b4c6_460d_8883_dd857554e019.slice/crio-293b3a7c3a43f146c6e97713c60c9db3f2668c0c6d5dc877d2d747bc8a0960e1 WatchSource:0}: Error finding container 293b3a7c3a43f146c6e97713c60c9db3f2668c0c6d5dc877d2d747bc8a0960e1: Status 404 returned error can't find the container with id 293b3a7c3a43f146c6e97713c60c9db3f2668c0c6d5dc877d2d747bc8a0960e1
May 26 11:25:41.224710 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.224685    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"]
May 26 11:25:41.228440 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.228416    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.232682 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.232481    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 26 11:25:41.232682 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.232517    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7lzxs\""
May 26 11:25:41.232831 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.232759    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 26 11:25:41.232831 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.232794    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 26 11:25:41.232971 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.232954    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-75ak8rrp360l\""
May 26 11:25:41.233072 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.233043    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 26 11:25:41.233170 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.233138    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 26 11:25:41.240519 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.240482    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"]
May 26 11:25:41.342752 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.342670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e736287-4f5d-4a35-b5c9-5124cffe8b06-metrics-client-ca\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.342752 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.342732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.343195 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.342771    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csch8\" (UniqueName: \"kubernetes.io/projected/2e736287-4f5d-4a35-b5c9-5124cffe8b06-kube-api-access-csch8\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.343195 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.342823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-thanos-querier-tls\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.343195 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.342874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.343195 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.342908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.343195 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.342933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-grpc-tls\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.343195 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.343016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.443971 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.443931    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e736287-4f5d-4a35-b5c9-5124cffe8b06-metrics-client-ca\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.443971 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.443978    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.444227 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.444012    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-csch8\" (UniqueName: \"kubernetes.io/projected/2e736287-4f5d-4a35-b5c9-5124cffe8b06-kube-api-access-csch8\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.444227 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.444049    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-thanos-querier-tls\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.444227 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.444101    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.444227 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.444142    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.444227 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.444168    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-grpc-tls\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.444227 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.444205    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.445616 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.445013    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e736287-4f5d-4a35-b5c9-5124cffe8b06-metrics-client-ca\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.447571 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.447350    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.447808 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.447782    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-thanos-querier-tls\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.449163 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.449111    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.449271 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.449190    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.449471 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.449451    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-grpc-tls\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.449533 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.449448    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e736287-4f5d-4a35-b5c9-5124cffe8b06-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.452762 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.452742    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csch8\" (UniqueName: \"kubernetes.io/projected/2e736287-4f5d-4a35-b5c9-5124cffe8b06-kube-api-access-csch8\") pod \"thanos-querier-f779f9ffb-4wxmj\" (UID: \"2e736287-4f5d-4a35-b5c9-5124cffe8b06\") " pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.541820 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.541780    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:41.943932 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:41.943664    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"]
May 26 11:25:41.946208 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:25:41.946176    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e736287_4f5d_4a35_b5c9_5124cffe8b06.slice/crio-bbc4f6e09ef7d2f1ff62687aacb14d9f7fd67767deebe73824b6e9a78d2bfa96 WatchSource:0}: Error finding container bbc4f6e09ef7d2f1ff62687aacb14d9f7fd67767deebe73824b6e9a78d2bfa96: Status 404 returned error can't find the container with id bbc4f6e09ef7d2f1ff62687aacb14d9f7fd67767deebe73824b6e9a78d2bfa96
May 26 11:25:42.123312 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:42.122448    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl" event={"ID":"93f480d3-f56c-4fa5-9ad4-dd2252a70d5a","Type":"ContainerStarted","Data":"7f4c3af3a14d448b7f8e5103e984b9d2fba7b9897b7483664027755b3b4739b3"}
May 26 11:25:42.128053 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:42.127926    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-949ct" event={"ID":"486de637-3719-41d0-bc8a-d1969384daa3","Type":"ContainerDied","Data":"5b809c325f27335ebe28347a6b748d324ac4af619f29f90ea7bae4fb1b2f14d1"}
May 26 11:25:42.128053 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:42.127905    2563 generic.go:358] "Generic (PLEG): container finished" podID="486de637-3719-41d0-bc8a-d1969384daa3" containerID="5b809c325f27335ebe28347a6b748d324ac4af619f29f90ea7bae4fb1b2f14d1" exitCode=0
May 26 11:25:42.130334 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:42.130263    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj" event={"ID":"2e736287-4f5d-4a35-b5c9-5124cffe8b06","Type":"ContainerStarted","Data":"bbc4f6e09ef7d2f1ff62687aacb14d9f7fd67767deebe73824b6e9a78d2bfa96"}
May 26 11:25:42.136705 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:42.136604    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9" event={"ID":"37858259-5496-44a7-984d-78eafc49f221","Type":"ContainerStarted","Data":"9a2ad8eb9de504ef037eb99f294f05eccf2b3c5aee8be61ab594b247b5e79deb"}
May 26 11:25:42.136705 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:42.136631    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9" event={"ID":"37858259-5496-44a7-984d-78eafc49f221","Type":"ContainerStarted","Data":"a2af0dc591b1789833d01b865785bcc8d559ce2ab592ce5779a89b8c866033eb"}
May 26 11:25:42.136705 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:42.136644    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9" event={"ID":"37858259-5496-44a7-984d-78eafc49f221","Type":"ContainerStarted","Data":"c59097c239150c3f0b83c09cab9f76c0d0fad9013ae11a4aea6f89ae84c2e5a5"}
May 26 11:25:42.138564 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:42.138378    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990dfe56-b4c6-460d-8883-dd857554e019","Type":"ContainerStarted","Data":"293b3a7c3a43f146c6e97713c60c9db3f2668c0c6d5dc877d2d747bc8a0960e1"}
May 26 11:25:42.164640 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:42.163454    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-dbfkl" podStartSLOduration=1.8008520159999999 podStartE2EDuration="3.163435897s" podCreationTimestamp="2026-05-26 11:25:39 +0000 UTC" firstStartedPulling="2026-05-26 11:25:39.688763004 +0000 UTC m=+72.494601446" lastFinishedPulling="2026-05-26 11:25:41.051346869 +0000 UTC m=+73.857185327" observedRunningTime="2026-05-26 11:25:42.144147752 +0000 UTC m=+74.949986217" watchObservedRunningTime="2026-05-26 11:25:42.163435897 +0000 UTC m=+74.969274363"
May 26 11:25:43.151392 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:43.151355    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-949ct" event={"ID":"486de637-3719-41d0-bc8a-d1969384daa3","Type":"ContainerStarted","Data":"cdb3e63d518fa2fc5a70fc357e9b22c49787c9b66c9c4a12d5f7f6a9e9f4bd7e"}
May 26 11:25:43.151905 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:43.151883    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-949ct" event={"ID":"486de637-3719-41d0-bc8a-d1969384daa3","Type":"ContainerStarted","Data":"3c959554f8fd889b6a81c925d182f374531f5acbdc3fa2ac084cef2e6b83ab97"}
May 26 11:25:43.171166 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:43.171116    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-949ct" podStartSLOduration=3.218044741 podStartE2EDuration="4.171100952s" podCreationTimestamp="2026-05-26 11:25:39 +0000 UTC" firstStartedPulling="2026-05-26 11:25:40.09871466 +0000 UTC m=+72.904553122" lastFinishedPulling="2026-05-26 11:25:41.051770875 +0000 UTC m=+73.857609333" observedRunningTime="2026-05-26 11:25:43.169793027 +0000 UTC m=+75.975631493" watchObservedRunningTime="2026-05-26 11:25:43.171100952 +0000 UTC m=+75.976939414"
May 26 11:25:43.172011 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:43.171969    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-277l9" podStartSLOduration=2.611711021 podStartE2EDuration="4.171960231s" podCreationTimestamp="2026-05-26 11:25:39 +0000 UTC" firstStartedPulling="2026-05-26 11:25:40.269467765 +0000 UTC m=+73.075306207" lastFinishedPulling="2026-05-26 11:25:41.829716971 +0000 UTC m=+74.635555417" observedRunningTime="2026-05-26 11:25:42.184217408 +0000 UTC m=+74.990055873" watchObservedRunningTime="2026-05-26 11:25:43.171960231 +0000 UTC m=+75.977798698"
May 26 11:25:45.159087 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:45.159055    2563 generic.go:358] "Generic (PLEG): container finished" podID="990dfe56-b4c6-460d-8883-dd857554e019" containerID="a3fcc405e77b9db74399e69b971f42b4e97a4211b5d923a4295628760ab1198d" exitCode=0
May 26 11:25:45.159517 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:45.159141    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990dfe56-b4c6-460d-8883-dd857554e019","Type":"ContainerDied","Data":"a3fcc405e77b9db74399e69b971f42b4e97a4211b5d923a4295628760ab1198d"}
May 26 11:25:45.160881 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:45.160744    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59cb6d89c7-qrjpj" event={"ID":"516e577e-045a-476d-a62d-16dc0c999a91","Type":"ContainerStarted","Data":"75c01442e798b4f238364452f4dc8359ab5fce1e2bc439525429aedcc5f87560"}
May 26 11:25:45.162719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:45.162700    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj" event={"ID":"2e736287-4f5d-4a35-b5c9-5124cffe8b06","Type":"ContainerStarted","Data":"909f01053509704e1b55336c9ecc42e4e247023e58d0fa095e29b0a322f0fab5"}
May 26 11:25:45.162814 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:45.162726    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj" event={"ID":"2e736287-4f5d-4a35-b5c9-5124cffe8b06","Type":"ContainerStarted","Data":"79e6b4158564ec875ca8f838c286db4dd3659a4cbc15dc3a24e3b8318b80b4b6"}
May 26 11:25:45.162814 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:45.162736    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj" event={"ID":"2e736287-4f5d-4a35-b5c9-5124cffe8b06","Type":"ContainerStarted","Data":"af21ef708548a4258e6578dd2b5cccd30622bd0e2731f7e7c3d3df91ba8f9ea0"}
May 26 11:25:45.203838 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:45.203786    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-59cb6d89c7-qrjpj" podStartSLOduration=1.989881569 podStartE2EDuration="6.203773132s" podCreationTimestamp="2026-05-26 11:25:39 +0000 UTC" firstStartedPulling="2026-05-26 11:25:40.245905062 +0000 UTC m=+73.051743505" lastFinishedPulling="2026-05-26 11:25:44.459796614 +0000 UTC m=+77.265635068" observedRunningTime="2026-05-26 11:25:45.202562106 +0000 UTC m=+78.008400601" watchObservedRunningTime="2026-05-26 11:25:45.203773132 +0000 UTC m=+78.009611597"
May 26 11:25:46.169787 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:46.169746    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj" event={"ID":"2e736287-4f5d-4a35-b5c9-5124cffe8b06","Type":"ContainerStarted","Data":"149889baf93097a87b1cc542cf36027ce3a86b53722dbafe027106695565a56c"}
May 26 11:25:46.169787 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:46.169794    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj" event={"ID":"2e736287-4f5d-4a35-b5c9-5124cffe8b06","Type":"ContainerStarted","Data":"234fac396d2713e3dff03ca9c2fd4ead51984b9c0d30605d3004c4fa85f90d85"}
May 26 11:25:46.170234 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:46.169809    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj" event={"ID":"2e736287-4f5d-4a35-b5c9-5124cffe8b06","Type":"ContainerStarted","Data":"0c57a2a8c5ccd4240d6d3e4871d612dc87a2ab54b45eed3c3b602a3895b74cfe"}
May 26 11:25:46.196693 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:46.196638    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj" podStartSLOduration=1.65179183 podStartE2EDuration="5.196617814s" podCreationTimestamp="2026-05-26 11:25:41 +0000 UTC" firstStartedPulling="2026-05-26 11:25:41.948803559 +0000 UTC m=+74.754642001" lastFinishedPulling="2026-05-26 11:25:45.493629544 +0000 UTC m=+78.299467985" observedRunningTime="2026-05-26 11:25:46.195134937 +0000 UTC m=+79.000973404" watchObservedRunningTime="2026-05-26 11:25:46.196617814 +0000 UTC m=+79.002456280"
May 26 11:25:47.174890 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:47.174851    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990dfe56-b4c6-460d-8883-dd857554e019","Type":"ContainerStarted","Data":"1df073f5cbc216ffc238cb9b73da4f6e28ccb4ed266ce8b3b80be5e6d303048d"}
May 26 11:25:47.174890 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:47.174888    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990dfe56-b4c6-460d-8883-dd857554e019","Type":"ContainerStarted","Data":"06ba9f2cabb87f0f465ec6ada9f24318e918ed3f532a4fc17d3aee36842e6a61"}
May 26 11:25:47.174890 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:47.174897    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990dfe56-b4c6-460d-8883-dd857554e019","Type":"ContainerStarted","Data":"ca7d98d51ae9a7f6d2b9ef216333bb4926433ac51346558787c02574153055e0"}
May 26 11:25:47.175442 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:47.174909    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990dfe56-b4c6-460d-8883-dd857554e019","Type":"ContainerStarted","Data":"326eb2f74a43274b29370bc396cd3623740ee6eb772dd6a220d7b4f81d4546aa"}
May 26 11:25:47.175442 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:47.174917    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990dfe56-b4c6-460d-8883-dd857554e019","Type":"ContainerStarted","Data":"d8250e577b0b96418acf65a4deb8ac13d247d87b80165d4b4247cd5414686697"}
May 26 11:25:47.175442 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:47.174927    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990dfe56-b4c6-460d-8883-dd857554e019","Type":"ContainerStarted","Data":"d5cf9df7fa42436145ea7401304731d46dc5ffbdcf34f554bcc08a56df894dc2"}
May 26 11:25:47.175592 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:47.175450    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:47.203881 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:47.203834    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.675554081 podStartE2EDuration="7.203819498s" podCreationTimestamp="2026-05-26 11:25:40 +0000 UTC" firstStartedPulling="2026-05-26 11:25:41.21569013 +0000 UTC m=+74.021528579" lastFinishedPulling="2026-05-26 11:25:46.74395555 +0000 UTC m=+79.549793996" observedRunningTime="2026-05-26 11:25:47.201612082 +0000 UTC m=+80.007450547" watchObservedRunningTime="2026-05-26 11:25:47.203819498 +0000 UTC m=+80.009657961"
May 26 11:25:47.459745 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:47.459667    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:50.076644 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:50.076603    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:50.077077 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:50.076681    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:50.081147 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:50.081125    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:50.189520 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:50.189489    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:25:51.173407 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.173369    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-65ddff8745-p5kb2"]
May 26 11:25:51.178754 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.178727    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.186986 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.186957    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 26 11:25:51.188947 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.188922    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65ddff8745-p5kb2"]
May 26 11:25:51.333641 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.333608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-serving-cert\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.333641 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.333647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-trusted-ca-bundle\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.333875 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.333668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-service-ca\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.333875 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.333809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-oauth-serving-cert\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.333875 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.333865    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-oauth-config\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.334019 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.333949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-config\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.334168 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.334130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhh84\" (UniqueName: \"kubernetes.io/projected/2909cd2e-db19-4b80-b408-34ecaaf971e9-kube-api-access-zhh84\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.434987 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.434907    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zhh84\" (UniqueName: \"kubernetes.io/projected/2909cd2e-db19-4b80-b408-34ecaaf971e9-kube-api-access-zhh84\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.434987 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.434948    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-serving-cert\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.434987 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.434967    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-trusted-ca-bundle\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.434987 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.434985    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-service-ca\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.435263 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.435024    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-oauth-serving-cert\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.435263 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.435051    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-oauth-config\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.435263 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.435076    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-config\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.435862 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.435833    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-oauth-serving-cert\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.435953 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.435877    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-config\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.435953 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.435894    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-trusted-ca-bundle\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.435953 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.435877    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-service-ca\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.437351 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.437327    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-oauth-config\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.437525 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.437506    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-serving-cert\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.444723 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.444700    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhh84\" (UniqueName: \"kubernetes.io/projected/2909cd2e-db19-4b80-b408-34ecaaf971e9-kube-api-access-zhh84\") pod \"console-65ddff8745-p5kb2\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") " pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.490531 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.490489    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:25:51.613766 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:51.613733    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65ddff8745-p5kb2"]
May 26 11:25:51.616962 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:25:51.616932    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2909cd2e_db19_4b80_b408_34ecaaf971e9.slice/crio-977319bf2bf79ff5cd920e617d2cf272bf6a53c05d0626aa60038024d6fbf15c WatchSource:0}: Error finding container 977319bf2bf79ff5cd920e617d2cf272bf6a53c05d0626aa60038024d6fbf15c: Status 404 returned error can't find the container with id 977319bf2bf79ff5cd920e617d2cf272bf6a53c05d0626aa60038024d6fbf15c
May 26 11:25:52.192725 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.192681    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65ddff8745-p5kb2" event={"ID":"2909cd2e-db19-4b80-b408-34ecaaf971e9","Type":"ContainerStarted","Data":"931f76bc7f2f826882a2d4504999881911fa9f16794147bc014ba0a76582247e"}
May 26 11:25:52.192725 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.192730    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65ddff8745-p5kb2" event={"ID":"2909cd2e-db19-4b80-b408-34ecaaf971e9","Type":"ContainerStarted","Data":"977319bf2bf79ff5cd920e617d2cf272bf6a53c05d0626aa60038024d6fbf15c"}
May 26 11:25:52.214938 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.214892    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-65ddff8745-p5kb2" podStartSLOduration=1.214874498 podStartE2EDuration="1.214874498s" podCreationTimestamp="2026-05-26 11:25:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:25:52.213667629 +0000 UTC m=+85.019506093" watchObservedRunningTime="2026-05-26 11:25:52.214874498 +0000 UTC m=+85.020712963"
May 26 11:25:52.473740 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.473647    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2" podUID="195ce04c-023e-47d6-b5f5-22b90196e4b2" containerName="registry" containerID="cri-o://4da899cba19066a77af721680ba06f57c2f496255972fdcb0efcd91c1948321b" gracePeriod=30
May 26 11:25:52.704631 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.704609    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:52.848801 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.848769    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/195ce04c-023e-47d6-b5f5-22b90196e4b2-image-registry-private-configuration\") pod \"195ce04c-023e-47d6-b5f5-22b90196e4b2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") "
May 26 11:25:52.848945 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.848812    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-certificates\") pod \"195ce04c-023e-47d6-b5f5-22b90196e4b2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") "
May 26 11:25:52.848945 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.848857    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/195ce04c-023e-47d6-b5f5-22b90196e4b2-installation-pull-secrets\") pod \"195ce04c-023e-47d6-b5f5-22b90196e4b2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") "
May 26 11:25:52.848945 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.848882    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls\") pod \"195ce04c-023e-47d6-b5f5-22b90196e4b2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") "
May 26 11:25:52.848945 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.848927    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/195ce04c-023e-47d6-b5f5-22b90196e4b2-ca-trust-extracted\") pod \"195ce04c-023e-47d6-b5f5-22b90196e4b2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") "
May 26 11:25:52.849125 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.848964    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/195ce04c-023e-47d6-b5f5-22b90196e4b2-trusted-ca\") pod \"195ce04c-023e-47d6-b5f5-22b90196e4b2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") "
May 26 11:25:52.849125 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.848991    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb8ps\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-kube-api-access-pb8ps\") pod \"195ce04c-023e-47d6-b5f5-22b90196e4b2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") "
May 26 11:25:52.849125 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.849047    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-bound-sa-token\") pod \"195ce04c-023e-47d6-b5f5-22b90196e4b2\" (UID: \"195ce04c-023e-47d6-b5f5-22b90196e4b2\") "
May 26 11:25:52.849415 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.849381    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "195ce04c-023e-47d6-b5f5-22b90196e4b2" (UID: "195ce04c-023e-47d6-b5f5-22b90196e4b2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:25:52.849712 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.849686    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195ce04c-023e-47d6-b5f5-22b90196e4b2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "195ce04c-023e-47d6-b5f5-22b90196e4b2" (UID: "195ce04c-023e-47d6-b5f5-22b90196e4b2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:25:52.851673 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.851650    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "195ce04c-023e-47d6-b5f5-22b90196e4b2" (UID: "195ce04c-023e-47d6-b5f5-22b90196e4b2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:25:52.851807 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.851783    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195ce04c-023e-47d6-b5f5-22b90196e4b2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "195ce04c-023e-47d6-b5f5-22b90196e4b2" (UID: "195ce04c-023e-47d6-b5f5-22b90196e4b2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:25:52.851911 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.851817    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "195ce04c-023e-47d6-b5f5-22b90196e4b2" (UID: "195ce04c-023e-47d6-b5f5-22b90196e4b2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:25:52.851911 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.851889    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195ce04c-023e-47d6-b5f5-22b90196e4b2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "195ce04c-023e-47d6-b5f5-22b90196e4b2" (UID: "195ce04c-023e-47d6-b5f5-22b90196e4b2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:25:52.851999 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.851981    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-kube-api-access-pb8ps" (OuterVolumeSpecName: "kube-api-access-pb8ps") pod "195ce04c-023e-47d6-b5f5-22b90196e4b2" (UID: "195ce04c-023e-47d6-b5f5-22b90196e4b2"). InnerVolumeSpecName "kube-api-access-pb8ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:25:52.857385 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.857363    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195ce04c-023e-47d6-b5f5-22b90196e4b2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "195ce04c-023e-47d6-b5f5-22b90196e4b2" (UID: "195ce04c-023e-47d6-b5f5-22b90196e4b2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:25:52.950638 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.950599    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-bound-sa-token\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:25:52.950638 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.950631    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/195ce04c-023e-47d6-b5f5-22b90196e4b2-image-registry-private-configuration\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:25:52.950638 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.950645    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-certificates\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:25:52.950864 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.950657    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/195ce04c-023e-47d6-b5f5-22b90196e4b2-installation-pull-secrets\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:25:52.950864 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.950669    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-registry-tls\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:25:52.950864 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.950680    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/195ce04c-023e-47d6-b5f5-22b90196e4b2-ca-trust-extracted\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:25:52.950864 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.950691    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/195ce04c-023e-47d6-b5f5-22b90196e4b2-trusted-ca\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:25:52.950864 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:52.950703    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb8ps\" (UniqueName: \"kubernetes.io/projected/195ce04c-023e-47d6-b5f5-22b90196e4b2-kube-api-access-pb8ps\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:25:53.183555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:53.183471    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-f779f9ffb-4wxmj"
May 26 11:25:53.196303 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:53.196253    2563 generic.go:358] "Generic (PLEG): container finished" podID="195ce04c-023e-47d6-b5f5-22b90196e4b2" containerID="4da899cba19066a77af721680ba06f57c2f496255972fdcb0efcd91c1948321b" exitCode=0
May 26 11:25:53.196724 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:53.196343    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2" event={"ID":"195ce04c-023e-47d6-b5f5-22b90196e4b2","Type":"ContainerDied","Data":"4da899cba19066a77af721680ba06f57c2f496255972fdcb0efcd91c1948321b"}
May 26 11:25:53.196724 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:53.196366    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2"
May 26 11:25:53.196724 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:53.196388    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74c679c6fb-mpsq2" event={"ID":"195ce04c-023e-47d6-b5f5-22b90196e4b2","Type":"ContainerDied","Data":"e180dfe953ca6f9c77eef0e84d92a85fbd341d512b324c8fdc1cdcac5c3befe8"}
May 26 11:25:53.196724 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:53.196423    2563 scope.go:117] "RemoveContainer" containerID="4da899cba19066a77af721680ba06f57c2f496255972fdcb0efcd91c1948321b"
May 26 11:25:53.204469 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:53.204451    2563 scope.go:117] "RemoveContainer" containerID="4da899cba19066a77af721680ba06f57c2f496255972fdcb0efcd91c1948321b"
May 26 11:25:53.204711 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:25:53.204694    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da899cba19066a77af721680ba06f57c2f496255972fdcb0efcd91c1948321b\": container with ID starting with 4da899cba19066a77af721680ba06f57c2f496255972fdcb0efcd91c1948321b not found: ID does not exist" containerID="4da899cba19066a77af721680ba06f57c2f496255972fdcb0efcd91c1948321b"
May 26 11:25:53.204761 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:53.204718    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da899cba19066a77af721680ba06f57c2f496255972fdcb0efcd91c1948321b"} err="failed to get container status \"4da899cba19066a77af721680ba06f57c2f496255972fdcb0efcd91c1948321b\": rpc error: code = NotFound desc = could not find container \"4da899cba19066a77af721680ba06f57c2f496255972fdcb0efcd91c1948321b\": container with ID starting with 4da899cba19066a77af721680ba06f57c2f496255972fdcb0efcd91c1948321b not found: ID does not exist"
May 26 11:25:53.223080 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:53.223055    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74c679c6fb-mpsq2"]
May 26 11:25:53.228096 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:53.228072    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74c679c6fb-mpsq2"]
May 26 11:25:53.797739 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:25:53.797703    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="195ce04c-023e-47d6-b5f5-22b90196e4b2" path="/var/lib/kubelet/pods/195ce04c-023e-47d6-b5f5-22b90196e4b2/volumes"
May 26 11:26:01.490832 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:01.490799    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:26:01.491318 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:01.490942    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:26:01.495613 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:01.495588    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:26:02.227404 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:02.227374    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:26:02.289440 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:02.289403    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59cb6d89c7-qrjpj"]
May 26 11:26:09.096209 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:09.096166    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5pwxn"
May 26 11:26:13.547464 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:13.547439    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rtrlh_3f9d7bb9-08d8-495f-9713-a6432e7f6a4f/serve-healthcheck-canary/0.log"
May 26 11:26:27.308718 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.308674    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-59cb6d89c7-qrjpj" podUID="516e577e-045a-476d-a62d-16dc0c999a91" containerName="console" containerID="cri-o://75c01442e798b4f238364452f4dc8359ab5fce1e2bc439525429aedcc5f87560" gracePeriod=15
May 26 11:26:27.575764 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.575741    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-59cb6d89c7-qrjpj_516e577e-045a-476d-a62d-16dc0c999a91/console/0.log"
May 26 11:26:27.575876 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.575806    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:26:27.646604 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.646572    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-console-config\") pod \"516e577e-045a-476d-a62d-16dc0c999a91\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") "
May 26 11:26:27.646767 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.646636    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/516e577e-045a-476d-a62d-16dc0c999a91-console-oauth-config\") pod \"516e577e-045a-476d-a62d-16dc0c999a91\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") "
May 26 11:26:27.646767 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.646677    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-oauth-serving-cert\") pod \"516e577e-045a-476d-a62d-16dc0c999a91\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") "
May 26 11:26:27.646767 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.646710    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-service-ca\") pod \"516e577e-045a-476d-a62d-16dc0c999a91\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") "
May 26 11:26:27.646889 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.646831    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc2b5\" (UniqueName: \"kubernetes.io/projected/516e577e-045a-476d-a62d-16dc0c999a91-kube-api-access-rc2b5\") pod \"516e577e-045a-476d-a62d-16dc0c999a91\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") "
May 26 11:26:27.646889 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.646873    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/516e577e-045a-476d-a62d-16dc0c999a91-console-serving-cert\") pod \"516e577e-045a-476d-a62d-16dc0c999a91\" (UID: \"516e577e-045a-476d-a62d-16dc0c999a91\") "
May 26 11:26:27.647057 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.647034    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-console-config" (OuterVolumeSpecName: "console-config") pod "516e577e-045a-476d-a62d-16dc0c999a91" (UID: "516e577e-045a-476d-a62d-16dc0c999a91"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:26:27.647131 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.647065    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-service-ca" (OuterVolumeSpecName: "service-ca") pod "516e577e-045a-476d-a62d-16dc0c999a91" (UID: "516e577e-045a-476d-a62d-16dc0c999a91"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:26:27.647181 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.647123    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "516e577e-045a-476d-a62d-16dc0c999a91" (UID: "516e577e-045a-476d-a62d-16dc0c999a91"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:26:27.647181 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.647137    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-service-ca\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:26:27.647250 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.647185    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-console-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:26:27.649019 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.648990    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/516e577e-045a-476d-a62d-16dc0c999a91-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "516e577e-045a-476d-a62d-16dc0c999a91" (UID: "516e577e-045a-476d-a62d-16dc0c999a91"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:26:27.649019 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.649006    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/516e577e-045a-476d-a62d-16dc0c999a91-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "516e577e-045a-476d-a62d-16dc0c999a91" (UID: "516e577e-045a-476d-a62d-16dc0c999a91"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:26:27.649181 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.649158    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/516e577e-045a-476d-a62d-16dc0c999a91-kube-api-access-rc2b5" (OuterVolumeSpecName: "kube-api-access-rc2b5") pod "516e577e-045a-476d-a62d-16dc0c999a91" (UID: "516e577e-045a-476d-a62d-16dc0c999a91"). InnerVolumeSpecName "kube-api-access-rc2b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:26:27.748100 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.748071    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/516e577e-045a-476d-a62d-16dc0c999a91-console-oauth-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:26:27.748100 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.748102    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/516e577e-045a-476d-a62d-16dc0c999a91-oauth-serving-cert\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:26:27.748275 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.748113    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc2b5\" (UniqueName: \"kubernetes.io/projected/516e577e-045a-476d-a62d-16dc0c999a91-kube-api-access-rc2b5\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:26:27.748275 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:27.748121    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/516e577e-045a-476d-a62d-16dc0c999a91-console-serving-cert\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:26:28.300585 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:28.300547    2563 generic.go:358] "Generic (PLEG): container finished" podID="516e577e-045a-476d-a62d-16dc0c999a91" containerID="75c01442e798b4f238364452f4dc8359ab5fce1e2bc439525429aedcc5f87560" exitCode=2
May 26 11:26:28.300731 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:28.300642    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59cb6d89c7-qrjpj"
May 26 11:26:28.300731 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:28.300640    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59cb6d89c7-qrjpj" event={"ID":"516e577e-045a-476d-a62d-16dc0c999a91","Type":"ContainerDied","Data":"75c01442e798b4f238364452f4dc8359ab5fce1e2bc439525429aedcc5f87560"}
May 26 11:26:28.300731 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:28.300686    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59cb6d89c7-qrjpj" event={"ID":"516e577e-045a-476d-a62d-16dc0c999a91","Type":"ContainerDied","Data":"9ea51cdaa85bca9024ebc6e5747f94ba065f873e90aeed2c8e51fb8e858a8e2b"}
May 26 11:26:28.300731 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:28.300720    2563 scope.go:117] "RemoveContainer" containerID="75c01442e798b4f238364452f4dc8359ab5fce1e2bc439525429aedcc5f87560"
May 26 11:26:28.307945 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:28.307930    2563 scope.go:117] "RemoveContainer" containerID="75c01442e798b4f238364452f4dc8359ab5fce1e2bc439525429aedcc5f87560"
May 26 11:26:28.308196 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:26:28.308174    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c01442e798b4f238364452f4dc8359ab5fce1e2bc439525429aedcc5f87560\": container with ID starting with 75c01442e798b4f238364452f4dc8359ab5fce1e2bc439525429aedcc5f87560 not found: ID does not exist" containerID="75c01442e798b4f238364452f4dc8359ab5fce1e2bc439525429aedcc5f87560"
May 26 11:26:28.308272 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:28.308218    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c01442e798b4f238364452f4dc8359ab5fce1e2bc439525429aedcc5f87560"} err="failed to get container status \"75c01442e798b4f238364452f4dc8359ab5fce1e2bc439525429aedcc5f87560\": rpc error: code = NotFound desc = could not find container \"75c01442e798b4f238364452f4dc8359ab5fce1e2bc439525429aedcc5f87560\": container with ID starting with 75c01442e798b4f238364452f4dc8359ab5fce1e2bc439525429aedcc5f87560 not found: ID does not exist"
May 26 11:26:28.330571 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:28.330539    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59cb6d89c7-qrjpj"]
May 26 11:26:28.332642 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:28.332617    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59cb6d89c7-qrjpj"]
May 26 11:26:29.796050 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:29.796012    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="516e577e-045a-476d-a62d-16dc0c999a91" path="/var/lib/kubelet/pods/516e577e-045a-476d-a62d-16dc0c999a91/volumes"
May 26 11:26:59.439488 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:59.439450    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:26:59.439958 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:59.439866    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="alertmanager" containerID="cri-o://d5cf9df7fa42436145ea7401304731d46dc5ffbdcf34f554bcc08a56df894dc2" gracePeriod=120
May 26 11:26:59.440040 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:59.439945    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="kube-rbac-proxy-metric" containerID="cri-o://06ba9f2cabb87f0f465ec6ada9f24318e918ed3f532a4fc17d3aee36842e6a61" gracePeriod=120
May 26 11:26:59.440040 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:59.439997    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="prom-label-proxy" containerID="cri-o://1df073f5cbc216ffc238cb9b73da4f6e28ccb4ed266ce8b3b80be5e6d303048d" gracePeriod=120
May 26 11:26:59.440145 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:59.440009    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="config-reloader" containerID="cri-o://d8250e577b0b96418acf65a4deb8ac13d247d87b80165d4b4247cd5414686697" gracePeriod=120
May 26 11:26:59.440145 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:59.439945    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="kube-rbac-proxy-web" containerID="cri-o://326eb2f74a43274b29370bc396cd3623740ee6eb772dd6a220d7b4f81d4546aa" gracePeriod=120
May 26 11:26:59.440145 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:26:59.439958    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="kube-rbac-proxy" containerID="cri-o://ca7d98d51ae9a7f6d2b9ef216333bb4926433ac51346558787c02574153055e0" gracePeriod=120
May 26 11:27:00.394868 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.394836    2563 generic.go:358] "Generic (PLEG): container finished" podID="990dfe56-b4c6-460d-8883-dd857554e019" containerID="1df073f5cbc216ffc238cb9b73da4f6e28ccb4ed266ce8b3b80be5e6d303048d" exitCode=0
May 26 11:27:00.394868 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.394863    2563 generic.go:358] "Generic (PLEG): container finished" podID="990dfe56-b4c6-460d-8883-dd857554e019" containerID="06ba9f2cabb87f0f465ec6ada9f24318e918ed3f532a4fc17d3aee36842e6a61" exitCode=0
May 26 11:27:00.394868 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.394869    2563 generic.go:358] "Generic (PLEG): container finished" podID="990dfe56-b4c6-460d-8883-dd857554e019" containerID="ca7d98d51ae9a7f6d2b9ef216333bb4926433ac51346558787c02574153055e0" exitCode=0
May 26 11:27:00.394868 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.394875    2563 generic.go:358] "Generic (PLEG): container finished" podID="990dfe56-b4c6-460d-8883-dd857554e019" containerID="d8250e577b0b96418acf65a4deb8ac13d247d87b80165d4b4247cd5414686697" exitCode=0
May 26 11:27:00.394868 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.394882    2563 generic.go:358] "Generic (PLEG): container finished" podID="990dfe56-b4c6-460d-8883-dd857554e019" containerID="d5cf9df7fa42436145ea7401304731d46dc5ffbdcf34f554bcc08a56df894dc2" exitCode=0
May 26 11:27:00.395159 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.394915    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990dfe56-b4c6-460d-8883-dd857554e019","Type":"ContainerDied","Data":"1df073f5cbc216ffc238cb9b73da4f6e28ccb4ed266ce8b3b80be5e6d303048d"}
May 26 11:27:00.395159 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.394962    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990dfe56-b4c6-460d-8883-dd857554e019","Type":"ContainerDied","Data":"06ba9f2cabb87f0f465ec6ada9f24318e918ed3f532a4fc17d3aee36842e6a61"}
May 26 11:27:00.395159 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.394978    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990dfe56-b4c6-460d-8883-dd857554e019","Type":"ContainerDied","Data":"ca7d98d51ae9a7f6d2b9ef216333bb4926433ac51346558787c02574153055e0"}
May 26 11:27:00.395159 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.394991    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990dfe56-b4c6-460d-8883-dd857554e019","Type":"ContainerDied","Data":"d8250e577b0b96418acf65a4deb8ac13d247d87b80165d4b4247cd5414686697"}
May 26 11:27:00.395159 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.395005    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990dfe56-b4c6-460d-8883-dd857554e019","Type":"ContainerDied","Data":"d5cf9df7fa42436145ea7401304731d46dc5ffbdcf34f554bcc08a56df894dc2"}
May 26 11:27:00.687087 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.687064    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:00.703039 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.703021    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b8dq\" (UniqueName: \"kubernetes.io/projected/990dfe56-b4c6-460d-8883-dd857554e019-kube-api-access-4b8dq\") pod \"990dfe56-b4c6-460d-8883-dd857554e019\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") "
May 26 11:27:00.703146 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.703050    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/990dfe56-b4c6-460d-8883-dd857554e019-tls-assets\") pod \"990dfe56-b4c6-460d-8883-dd857554e019\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") "
May 26 11:27:00.703146 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.703070    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/990dfe56-b4c6-460d-8883-dd857554e019-alertmanager-trusted-ca-bundle\") pod \"990dfe56-b4c6-460d-8883-dd857554e019\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") "
May 26 11:27:00.703146 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.703092    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-web-config\") pod \"990dfe56-b4c6-460d-8883-dd857554e019\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") "
May 26 11:27:00.703146 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.703117    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy-metric\") pod \"990dfe56-b4c6-460d-8883-dd857554e019\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") "
May 26 11:27:00.703363 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.703146    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy\") pod \"990dfe56-b4c6-460d-8883-dd857554e019\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") "
May 26 11:27:00.703363 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.703181    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/990dfe56-b4c6-460d-8883-dd857554e019-config-out\") pod \"990dfe56-b4c6-460d-8883-dd857554e019\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") "
May 26 11:27:00.703363 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.703204    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/990dfe56-b4c6-460d-8883-dd857554e019-alertmanager-main-db\") pod \"990dfe56-b4c6-460d-8883-dd857554e019\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") "
May 26 11:27:00.703363 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.703231    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-main-tls\") pod \"990dfe56-b4c6-460d-8883-dd857554e019\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") "
May 26 11:27:00.703363 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.703276    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-config-volume\") pod \"990dfe56-b4c6-460d-8883-dd857554e019\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") "
May 26 11:27:00.703363 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.703324    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy-web\") pod \"990dfe56-b4c6-460d-8883-dd857554e019\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") "
May 26 11:27:00.703363 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.703341    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/990dfe56-b4c6-460d-8883-dd857554e019-metrics-client-ca\") pod \"990dfe56-b4c6-460d-8883-dd857554e019\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") "
May 26 11:27:00.703363 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.703357    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-cluster-tls-config\") pod \"990dfe56-b4c6-460d-8883-dd857554e019\" (UID: \"990dfe56-b4c6-460d-8883-dd857554e019\") "
May 26 11:27:00.704571 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.703889    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990dfe56-b4c6-460d-8883-dd857554e019-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "990dfe56-b4c6-460d-8883-dd857554e019" (UID: "990dfe56-b4c6-460d-8883-dd857554e019"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:27:00.704571 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.704073    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990dfe56-b4c6-460d-8883-dd857554e019-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "990dfe56-b4c6-460d-8883-dd857554e019" (UID: "990dfe56-b4c6-460d-8883-dd857554e019"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:27:00.704571 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.704536    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990dfe56-b4c6-460d-8883-dd857554e019-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "990dfe56-b4c6-460d-8883-dd857554e019" (UID: "990dfe56-b4c6-460d-8883-dd857554e019"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:27:00.706249 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.706223    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-config-volume" (OuterVolumeSpecName: "config-volume") pod "990dfe56-b4c6-460d-8883-dd857554e019" (UID: "990dfe56-b4c6-460d-8883-dd857554e019"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:00.706904 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.706863    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "990dfe56-b4c6-460d-8883-dd857554e019" (UID: "990dfe56-b4c6-460d-8883-dd857554e019"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:00.706998 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.706910    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990dfe56-b4c6-460d-8883-dd857554e019-kube-api-access-4b8dq" (OuterVolumeSpecName: "kube-api-access-4b8dq") pod "990dfe56-b4c6-460d-8883-dd857554e019" (UID: "990dfe56-b4c6-460d-8883-dd857554e019"). InnerVolumeSpecName "kube-api-access-4b8dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:27:00.707332 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.707305    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "990dfe56-b4c6-460d-8883-dd857554e019" (UID: "990dfe56-b4c6-460d-8883-dd857554e019"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:00.707570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.707536    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990dfe56-b4c6-460d-8883-dd857554e019-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "990dfe56-b4c6-460d-8883-dd857554e019" (UID: "990dfe56-b4c6-460d-8883-dd857554e019"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:27:00.707570 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.707555    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "990dfe56-b4c6-460d-8883-dd857554e019" (UID: "990dfe56-b4c6-460d-8883-dd857554e019"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:00.707741 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.707719    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "990dfe56-b4c6-460d-8883-dd857554e019" (UID: "990dfe56-b4c6-460d-8883-dd857554e019"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:00.707798 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.707722    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990dfe56-b4c6-460d-8883-dd857554e019-config-out" (OuterVolumeSpecName: "config-out") pod "990dfe56-b4c6-460d-8883-dd857554e019" (UID: "990dfe56-b4c6-460d-8883-dd857554e019"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:27:00.711252 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.711227    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "990dfe56-b4c6-460d-8883-dd857554e019" (UID: "990dfe56-b4c6-460d-8883-dd857554e019"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:00.719210 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.719190    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-web-config" (OuterVolumeSpecName: "web-config") pod "990dfe56-b4c6-460d-8883-dd857554e019" (UID: "990dfe56-b4c6-460d-8883-dd857554e019"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:00.804657 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.804622    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/990dfe56-b4c6-460d-8883-dd857554e019-config-out\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:00.804657 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.804648    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/990dfe56-b4c6-460d-8883-dd857554e019-alertmanager-main-db\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:00.804657 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.804657    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-main-tls\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:00.804859 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.804667    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-config-volume\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:00.804859 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.804676    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:00.804859 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.804684    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/990dfe56-b4c6-460d-8883-dd857554e019-metrics-client-ca\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:00.804859 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.804692    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-cluster-tls-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:00.804859 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.804701    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b8dq\" (UniqueName: \"kubernetes.io/projected/990dfe56-b4c6-460d-8883-dd857554e019-kube-api-access-4b8dq\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:00.804859 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.804710    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/990dfe56-b4c6-460d-8883-dd857554e019-tls-assets\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:00.804859 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.804721    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/990dfe56-b4c6-460d-8883-dd857554e019-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:00.804859 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.804731    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-web-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:00.804859 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.804739    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:00.804859 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:00.804749    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/990dfe56-b4c6-460d-8883-dd857554e019-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:01.404499 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.404464    2563 generic.go:358] "Generic (PLEG): container finished" podID="990dfe56-b4c6-460d-8883-dd857554e019" containerID="326eb2f74a43274b29370bc396cd3623740ee6eb772dd6a220d7b4f81d4546aa" exitCode=0
May 26 11:27:01.404671 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.404536    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990dfe56-b4c6-460d-8883-dd857554e019","Type":"ContainerDied","Data":"326eb2f74a43274b29370bc396cd3623740ee6eb772dd6a220d7b4f81d4546aa"}
May 26 11:27:01.404671 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.404564    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990dfe56-b4c6-460d-8883-dd857554e019","Type":"ContainerDied","Data":"293b3a7c3a43f146c6e97713c60c9db3f2668c0c6d5dc877d2d747bc8a0960e1"}
May 26 11:27:01.404671 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.404579    2563 scope.go:117] "RemoveContainer" containerID="1df073f5cbc216ffc238cb9b73da4f6e28ccb4ed266ce8b3b80be5e6d303048d"
May 26 11:27:01.404671 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.404577    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.412357 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.412338    2563 scope.go:117] "RemoveContainer" containerID="06ba9f2cabb87f0f465ec6ada9f24318e918ed3f532a4fc17d3aee36842e6a61"
May 26 11:27:01.419131 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.419113    2563 scope.go:117] "RemoveContainer" containerID="ca7d98d51ae9a7f6d2b9ef216333bb4926433ac51346558787c02574153055e0"
May 26 11:27:01.425014 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.425000    2563 scope.go:117] "RemoveContainer" containerID="326eb2f74a43274b29370bc396cd3623740ee6eb772dd6a220d7b4f81d4546aa"
May 26 11:27:01.427977 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.427960    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:27:01.432034 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.432011    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:27:01.432853 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.432836    2563 scope.go:117] "RemoveContainer" containerID="d8250e577b0b96418acf65a4deb8ac13d247d87b80165d4b4247cd5414686697"
May 26 11:27:01.438786 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.438771    2563 scope.go:117] "RemoveContainer" containerID="d5cf9df7fa42436145ea7401304731d46dc5ffbdcf34f554bcc08a56df894dc2"
May 26 11:27:01.444809 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.444781    2563 scope.go:117] "RemoveContainer" containerID="a3fcc405e77b9db74399e69b971f42b4e97a4211b5d923a4295628760ab1198d"
May 26 11:27:01.450781 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.450760    2563 scope.go:117] "RemoveContainer" containerID="1df073f5cbc216ffc238cb9b73da4f6e28ccb4ed266ce8b3b80be5e6d303048d"
May 26 11:27:01.451168 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:27:01.451149    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df073f5cbc216ffc238cb9b73da4f6e28ccb4ed266ce8b3b80be5e6d303048d\": container with ID starting with 1df073f5cbc216ffc238cb9b73da4f6e28ccb4ed266ce8b3b80be5e6d303048d not found: ID does not exist" containerID="1df073f5cbc216ffc238cb9b73da4f6e28ccb4ed266ce8b3b80be5e6d303048d"
May 26 11:27:01.451242 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.451174    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df073f5cbc216ffc238cb9b73da4f6e28ccb4ed266ce8b3b80be5e6d303048d"} err="failed to get container status \"1df073f5cbc216ffc238cb9b73da4f6e28ccb4ed266ce8b3b80be5e6d303048d\": rpc error: code = NotFound desc = could not find container \"1df073f5cbc216ffc238cb9b73da4f6e28ccb4ed266ce8b3b80be5e6d303048d\": container with ID starting with 1df073f5cbc216ffc238cb9b73da4f6e28ccb4ed266ce8b3b80be5e6d303048d not found: ID does not exist"
May 26 11:27:01.451242 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.451192    2563 scope.go:117] "RemoveContainer" containerID="06ba9f2cabb87f0f465ec6ada9f24318e918ed3f532a4fc17d3aee36842e6a61"
May 26 11:27:01.451424 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:27:01.451409    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ba9f2cabb87f0f465ec6ada9f24318e918ed3f532a4fc17d3aee36842e6a61\": container with ID starting with 06ba9f2cabb87f0f465ec6ada9f24318e918ed3f532a4fc17d3aee36842e6a61 not found: ID does not exist" containerID="06ba9f2cabb87f0f465ec6ada9f24318e918ed3f532a4fc17d3aee36842e6a61"
May 26 11:27:01.451475 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.451428    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ba9f2cabb87f0f465ec6ada9f24318e918ed3f532a4fc17d3aee36842e6a61"} err="failed to get container status \"06ba9f2cabb87f0f465ec6ada9f24318e918ed3f532a4fc17d3aee36842e6a61\": rpc error: code = NotFound desc = could not find container \"06ba9f2cabb87f0f465ec6ada9f24318e918ed3f532a4fc17d3aee36842e6a61\": container with ID starting with 06ba9f2cabb87f0f465ec6ada9f24318e918ed3f532a4fc17d3aee36842e6a61 not found: ID does not exist"
May 26 11:27:01.451475 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.451442    2563 scope.go:117] "RemoveContainer" containerID="ca7d98d51ae9a7f6d2b9ef216333bb4926433ac51346558787c02574153055e0"
May 26 11:27:01.451692 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:27:01.451675    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca7d98d51ae9a7f6d2b9ef216333bb4926433ac51346558787c02574153055e0\": container with ID starting with ca7d98d51ae9a7f6d2b9ef216333bb4926433ac51346558787c02574153055e0 not found: ID does not exist" containerID="ca7d98d51ae9a7f6d2b9ef216333bb4926433ac51346558787c02574153055e0"
May 26 11:27:01.451752 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.451702    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7d98d51ae9a7f6d2b9ef216333bb4926433ac51346558787c02574153055e0"} err="failed to get container status \"ca7d98d51ae9a7f6d2b9ef216333bb4926433ac51346558787c02574153055e0\": rpc error: code = NotFound desc = could not find container \"ca7d98d51ae9a7f6d2b9ef216333bb4926433ac51346558787c02574153055e0\": container with ID starting with ca7d98d51ae9a7f6d2b9ef216333bb4926433ac51346558787c02574153055e0 not found: ID does not exist"
May 26 11:27:01.451752 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.451728    2563 scope.go:117] "RemoveContainer" containerID="326eb2f74a43274b29370bc396cd3623740ee6eb772dd6a220d7b4f81d4546aa"
May 26 11:27:01.451979 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:27:01.451964    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"326eb2f74a43274b29370bc396cd3623740ee6eb772dd6a220d7b4f81d4546aa\": container with ID starting with 326eb2f74a43274b29370bc396cd3623740ee6eb772dd6a220d7b4f81d4546aa not found: ID does not exist" containerID="326eb2f74a43274b29370bc396cd3623740ee6eb772dd6a220d7b4f81d4546aa"
May 26 11:27:01.452026 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.451983    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"326eb2f74a43274b29370bc396cd3623740ee6eb772dd6a220d7b4f81d4546aa"} err="failed to get container status \"326eb2f74a43274b29370bc396cd3623740ee6eb772dd6a220d7b4f81d4546aa\": rpc error: code = NotFound desc = could not find container \"326eb2f74a43274b29370bc396cd3623740ee6eb772dd6a220d7b4f81d4546aa\": container with ID starting with 326eb2f74a43274b29370bc396cd3623740ee6eb772dd6a220d7b4f81d4546aa not found: ID does not exist"
May 26 11:27:01.452026 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.452000    2563 scope.go:117] "RemoveContainer" containerID="d8250e577b0b96418acf65a4deb8ac13d247d87b80165d4b4247cd5414686697"
May 26 11:27:01.452223 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:27:01.452204    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8250e577b0b96418acf65a4deb8ac13d247d87b80165d4b4247cd5414686697\": container with ID starting with d8250e577b0b96418acf65a4deb8ac13d247d87b80165d4b4247cd5414686697 not found: ID does not exist" containerID="d8250e577b0b96418acf65a4deb8ac13d247d87b80165d4b4247cd5414686697"
May 26 11:27:01.452260 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.452230    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8250e577b0b96418acf65a4deb8ac13d247d87b80165d4b4247cd5414686697"} err="failed to get container status \"d8250e577b0b96418acf65a4deb8ac13d247d87b80165d4b4247cd5414686697\": rpc error: code = NotFound desc = could not find container \"d8250e577b0b96418acf65a4deb8ac13d247d87b80165d4b4247cd5414686697\": container with ID starting with d8250e577b0b96418acf65a4deb8ac13d247d87b80165d4b4247cd5414686697 not found: ID does not exist"
May 26 11:27:01.452260 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.452246    2563 scope.go:117] "RemoveContainer" containerID="d5cf9df7fa42436145ea7401304731d46dc5ffbdcf34f554bcc08a56df894dc2"
May 26 11:27:01.452583 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:27:01.452501    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5cf9df7fa42436145ea7401304731d46dc5ffbdcf34f554bcc08a56df894dc2\": container with ID starting with d5cf9df7fa42436145ea7401304731d46dc5ffbdcf34f554bcc08a56df894dc2 not found: ID does not exist" containerID="d5cf9df7fa42436145ea7401304731d46dc5ffbdcf34f554bcc08a56df894dc2"
May 26 11:27:01.452583 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.452518    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5cf9df7fa42436145ea7401304731d46dc5ffbdcf34f554bcc08a56df894dc2"} err="failed to get container status \"d5cf9df7fa42436145ea7401304731d46dc5ffbdcf34f554bcc08a56df894dc2\": rpc error: code = NotFound desc = could not find container \"d5cf9df7fa42436145ea7401304731d46dc5ffbdcf34f554bcc08a56df894dc2\": container with ID starting with d5cf9df7fa42436145ea7401304731d46dc5ffbdcf34f554bcc08a56df894dc2 not found: ID does not exist"
May 26 11:27:01.452583 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.452531    2563 scope.go:117] "RemoveContainer" containerID="a3fcc405e77b9db74399e69b971f42b4e97a4211b5d923a4295628760ab1198d"
May 26 11:27:01.452728 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:27:01.452711    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3fcc405e77b9db74399e69b971f42b4e97a4211b5d923a4295628760ab1198d\": container with ID starting with a3fcc405e77b9db74399e69b971f42b4e97a4211b5d923a4295628760ab1198d not found: ID does not exist" containerID="a3fcc405e77b9db74399e69b971f42b4e97a4211b5d923a4295628760ab1198d"
May 26 11:27:01.452766 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.452733    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3fcc405e77b9db74399e69b971f42b4e97a4211b5d923a4295628760ab1198d"} err="failed to get container status \"a3fcc405e77b9db74399e69b971f42b4e97a4211b5d923a4295628760ab1198d\": rpc error: code = NotFound desc = could not find container \"a3fcc405e77b9db74399e69b971f42b4e97a4211b5d923a4295628760ab1198d\": container with ID starting with a3fcc405e77b9db74399e69b971f42b4e97a4211b5d923a4295628760ab1198d not found: ID does not exist"
May 26 11:27:01.460698 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.460677    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:27:01.460994 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.460980    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="kube-rbac-proxy-metric"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.460997    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="kube-rbac-proxy-metric"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461012    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="kube-rbac-proxy"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461018    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="kube-rbac-proxy"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461027    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="kube-rbac-proxy-web"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461033    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="kube-rbac-proxy-web"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461042    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="prom-label-proxy"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461046    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="prom-label-proxy"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461060    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="init-config-reloader"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461064    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="init-config-reloader"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461071    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="alertmanager"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461078    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="alertmanager"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461087    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195ce04c-023e-47d6-b5f5-22b90196e4b2" containerName="registry"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461093    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="195ce04c-023e-47d6-b5f5-22b90196e4b2" containerName="registry"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461099    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="516e577e-045a-476d-a62d-16dc0c999a91" containerName="console"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461103    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="516e577e-045a-476d-a62d-16dc0c999a91" containerName="console"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461109    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="config-reloader"
May 26 11:27:01.461148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461114    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="config-reloader"
May 26 11:27:01.461660 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461157    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="516e577e-045a-476d-a62d-16dc0c999a91" containerName="console"
May 26 11:27:01.461660 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461168    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="alertmanager"
May 26 11:27:01.461660 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461175    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="195ce04c-023e-47d6-b5f5-22b90196e4b2" containerName="registry"
May 26 11:27:01.461660 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461182    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="config-reloader"
May 26 11:27:01.461660 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461188    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="kube-rbac-proxy-metric"
May 26 11:27:01.461660 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461194    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="kube-rbac-proxy-web"
May 26 11:27:01.461660 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461199    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="prom-label-proxy"
May 26 11:27:01.461660 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.461204    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="990dfe56-b4c6-460d-8883-dd857554e019" containerName="kube-rbac-proxy"
May 26 11:27:01.464700 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.464685    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.467193 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.467170    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 26 11:27:01.467313 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.467238    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 26 11:27:01.467313 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.467253    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 26 11:27:01.467438 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.467356    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 26 11:27:01.467438 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.467382    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 26 11:27:01.467438 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.467390    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hg54s\""
May 26 11:27:01.467438 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.467374    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 26 11:27:01.467592 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.467395    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 26 11:27:01.467640 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.467619    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 26 11:27:01.473335 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.473316    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 26 11:27:01.477462 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.477440    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:27:01.509784 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.509752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbgv5\" (UniqueName: \"kubernetes.io/projected/893717db-1c0f-4a5c-b9d0-b0fb328039e6-kube-api-access-cbgv5\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.509916 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.509797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-config-volume\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.509916 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.509855    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/893717db-1c0f-4a5c-b9d0-b0fb328039e6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.509916 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.509887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.509916 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.509912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/893717db-1c0f-4a5c-b9d0-b0fb328039e6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.510056 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.509969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/893717db-1c0f-4a5c-b9d0-b0fb328039e6-config-out\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.510056 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.510018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.510056 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.510050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.510157 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.510074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.510157 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.510109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-web-config\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.510157 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.510127    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.510157 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.510142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/893717db-1c0f-4a5c-b9d0-b0fb328039e6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.510493 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.510159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/893717db-1c0f-4a5c-b9d0-b0fb328039e6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.611169 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.611125    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.611169 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.611178    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-web-config\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.611411 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.611214    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.611411 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.611233    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/893717db-1c0f-4a5c-b9d0-b0fb328039e6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.611411 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.611252    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/893717db-1c0f-4a5c-b9d0-b0fb328039e6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.611411 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.611302    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cbgv5\" (UniqueName: \"kubernetes.io/projected/893717db-1c0f-4a5c-b9d0-b0fb328039e6-kube-api-access-cbgv5\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.611411 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.611329    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-config-volume\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.611411 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.611361    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/893717db-1c0f-4a5c-b9d0-b0fb328039e6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.611411 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.611399    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.611778 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.611426    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/893717db-1c0f-4a5c-b9d0-b0fb328039e6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.611778 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.611474    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/893717db-1c0f-4a5c-b9d0-b0fb328039e6-config-out\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.611778 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.611500    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.611778 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.611532    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.611977 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.611959    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/893717db-1c0f-4a5c-b9d0-b0fb328039e6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.612485 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.612461    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/893717db-1c0f-4a5c-b9d0-b0fb328039e6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.612594 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.612513    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/893717db-1c0f-4a5c-b9d0-b0fb328039e6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.614233 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.614186    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.614354 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.614239    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/893717db-1c0f-4a5c-b9d0-b0fb328039e6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.614354 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.614339    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.614510 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.614484    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-config-volume\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.615081 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.615053    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-web-config\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.615154 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.615107    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/893717db-1c0f-4a5c-b9d0-b0fb328039e6-config-out\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.615321 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.615273    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.615553 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.615532    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.615869 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.615850    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/893717db-1c0f-4a5c-b9d0-b0fb328039e6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.619876 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.619856    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbgv5\" (UniqueName: \"kubernetes.io/projected/893717db-1c0f-4a5c-b9d0-b0fb328039e6-kube-api-access-cbgv5\") pod \"alertmanager-main-0\" (UID: \"893717db-1c0f-4a5c-b9d0-b0fb328039e6\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.773922 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.773888    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 11:27:01.796304 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.796263    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="990dfe56-b4c6-460d-8883-dd857554e019" path="/var/lib/kubelet/pods/990dfe56-b4c6-460d-8883-dd857554e019/volumes"
May 26 11:27:01.915263 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:01.915228    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:27:01.918051 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:27:01.918020    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893717db_1c0f_4a5c_b9d0_b0fb328039e6.slice/crio-e4a0400b0af63f717b07355efba752275cde2c1dbe87a4bd22c4ebd766d2e0b1 WatchSource:0}: Error finding container e4a0400b0af63f717b07355efba752275cde2c1dbe87a4bd22c4ebd766d2e0b1: Status 404 returned error can't find the container with id e4a0400b0af63f717b07355efba752275cde2c1dbe87a4bd22c4ebd766d2e0b1
May 26 11:27:02.409699 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:02.409665    2563 generic.go:358] "Generic (PLEG): container finished" podID="893717db-1c0f-4a5c-b9d0-b0fb328039e6" containerID="d91decc93794097167febc5429756e503dc5a147de8f994ba7a8835ade53aa4a" exitCode=0
May 26 11:27:02.409875 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:02.409721    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"893717db-1c0f-4a5c-b9d0-b0fb328039e6","Type":"ContainerDied","Data":"d91decc93794097167febc5429756e503dc5a147de8f994ba7a8835ade53aa4a"}
May 26 11:27:02.409875 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:02.409756    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"893717db-1c0f-4a5c-b9d0-b0fb328039e6","Type":"ContainerStarted","Data":"e4a0400b0af63f717b07355efba752275cde2c1dbe87a4bd22c4ebd766d2e0b1"}
May 26 11:27:03.415351 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.415314    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"893717db-1c0f-4a5c-b9d0-b0fb328039e6","Type":"ContainerStarted","Data":"d8eca526785c55fb0f0b88f0d59ef04f003ca34dac67231bc5e7b885dfe836cc"}
May 26 11:27:03.415351 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.415353    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"893717db-1c0f-4a5c-b9d0-b0fb328039e6","Type":"ContainerStarted","Data":"961c827dbda32d5f981ee3cb7af219cccb0fbba79ba89a8ff4dbc3c880040935"}
May 26 11:27:03.415351 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.415363    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"893717db-1c0f-4a5c-b9d0-b0fb328039e6","Type":"ContainerStarted","Data":"575cafb426b3937d681117b927693d005c8c5e940ac389f26affa2e2cfe4ea4b"}
May 26 11:27:03.415833 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.415372    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"893717db-1c0f-4a5c-b9d0-b0fb328039e6","Type":"ContainerStarted","Data":"5f229ba6f522e087000e7d1849a04a9a4fc577a28c7af35835fea5159de4b080"}
May 26 11:27:03.415833 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.415380    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"893717db-1c0f-4a5c-b9d0-b0fb328039e6","Type":"ContainerStarted","Data":"8ddb0dc5cafcd5adcf8fea90973331f6e2b2717212e987fd79a7e3da7d03d35b"}
May 26 11:27:03.415833 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.415388    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"893717db-1c0f-4a5c-b9d0-b0fb328039e6","Type":"ContainerStarted","Data":"096180bc25af54609e1a8a44dfcfb511913a9456ff81a3d36a2d7f852b4f9a66"}
May 26 11:27:03.453984 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.453932    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.453917793 podStartE2EDuration="2.453917793s" podCreationTimestamp="2026-05-26 11:27:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:27:03.452508702 +0000 UTC m=+156.258347160" watchObservedRunningTime="2026-05-26 11:27:03.453917793 +0000 UTC m=+156.259756256"
May 26 11:27:03.514934 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.514898    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"]
May 26 11:27:03.517378 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.517360    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.519600 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.519574    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 26 11:27:03.519706 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.519619    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 26 11:27:03.519706 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.519674    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 26 11:27:03.519811 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.519677    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 26 11:27:03.519811 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.519709    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 26 11:27:03.519973 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.519958    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-nxzhs\""
May 26 11:27:03.524598 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.524578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c4256857-00c8-400a-b4bf-b58d10ed11c9-federate-client-tls\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.524670 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.524610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4256857-00c8-400a-b4bf-b58d10ed11c9-serving-certs-ca-bundle\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.524670 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.524636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4256857-00c8-400a-b4bf-b58d10ed11c9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.524670 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.524659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7s6n\" (UniqueName: \"kubernetes.io/projected/c4256857-00c8-400a-b4bf-b58d10ed11c9-kube-api-access-k7s6n\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.524782 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.524679    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c4256857-00c8-400a-b4bf-b58d10ed11c9-telemeter-client-tls\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.524782 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.524697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4256857-00c8-400a-b4bf-b58d10ed11c9-metrics-client-ca\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.524782 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.524735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c4256857-00c8-400a-b4bf-b58d10ed11c9-secret-telemeter-client\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.524782 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.524756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4256857-00c8-400a-b4bf-b58d10ed11c9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.526925 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.526909    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 26 11:27:03.530764 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.530739    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"]
May 26 11:27:03.625582 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.625538    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c4256857-00c8-400a-b4bf-b58d10ed11c9-federate-client-tls\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.625582 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.625583    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4256857-00c8-400a-b4bf-b58d10ed11c9-serving-certs-ca-bundle\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.625833 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.625620    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4256857-00c8-400a-b4bf-b58d10ed11c9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.625833 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.625648    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k7s6n\" (UniqueName: \"kubernetes.io/projected/c4256857-00c8-400a-b4bf-b58d10ed11c9-kube-api-access-k7s6n\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.625833 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.625672    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c4256857-00c8-400a-b4bf-b58d10ed11c9-telemeter-client-tls\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.625833 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.625697    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4256857-00c8-400a-b4bf-b58d10ed11c9-metrics-client-ca\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.625833 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.625727    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c4256857-00c8-400a-b4bf-b58d10ed11c9-secret-telemeter-client\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.625833 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.625754    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4256857-00c8-400a-b4bf-b58d10ed11c9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.626817 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.626788    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4256857-00c8-400a-b4bf-b58d10ed11c9-metrics-client-ca\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.626921 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.626879    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4256857-00c8-400a-b4bf-b58d10ed11c9-serving-certs-ca-bundle\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.626921 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.626884    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4256857-00c8-400a-b4bf-b58d10ed11c9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.628213 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.628185    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4256857-00c8-400a-b4bf-b58d10ed11c9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.628339 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.628319    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c4256857-00c8-400a-b4bf-b58d10ed11c9-secret-telemeter-client\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.628477 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.628461    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c4256857-00c8-400a-b4bf-b58d10ed11c9-telemeter-client-tls\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.628606 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.628587    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c4256857-00c8-400a-b4bf-b58d10ed11c9-federate-client-tls\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.633469 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.633448    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7s6n\" (UniqueName: \"kubernetes.io/projected/c4256857-00c8-400a-b4bf-b58d10ed11c9-kube-api-access-k7s6n\") pod \"telemeter-client-6bf6bb686f-67pvg\" (UID: \"c4256857-00c8-400a-b4bf-b58d10ed11c9\") " pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.827201 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.827163    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"
May 26 11:27:03.966957 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:03.966930    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6bf6bb686f-67pvg"]
May 26 11:27:03.969216 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:27:03.969182    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4256857_00c8_400a_b4bf_b58d10ed11c9.slice/crio-cd48731eedc18b6c1af6c4207e9468ca2aee4f060aa0f4e30f55d70a91d16ada WatchSource:0}: Error finding container cd48731eedc18b6c1af6c4207e9468ca2aee4f060aa0f4e30f55d70a91d16ada: Status 404 returned error can't find the container with id cd48731eedc18b6c1af6c4207e9468ca2aee4f060aa0f4e30f55d70a91d16ada
May 26 11:27:04.418943 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:04.418907    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg" event={"ID":"c4256857-00c8-400a-b4bf-b58d10ed11c9","Type":"ContainerStarted","Data":"cd48731eedc18b6c1af6c4207e9468ca2aee4f060aa0f4e30f55d70a91d16ada"}
May 26 11:27:06.427598 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:06.427563    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg" event={"ID":"c4256857-00c8-400a-b4bf-b58d10ed11c9","Type":"ContainerStarted","Data":"c6302be4493e3a192d180a679d734753ac72b365277301505ae32186bd4924a4"}
May 26 11:27:06.427598 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:06.427603    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg" event={"ID":"c4256857-00c8-400a-b4bf-b58d10ed11c9","Type":"ContainerStarted","Data":"89fe114228cc7b97f5720319dbfa37b36995bad0eda1bbd667d2ca2bfc250073"}
May 26 11:27:06.428024 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:06.427616    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg" event={"ID":"c4256857-00c8-400a-b4bf-b58d10ed11c9","Type":"ContainerStarted","Data":"fc2248a004d8665cdbc528608731df49ba111aecde239049f1eec07b28676b74"}
May 26 11:27:06.449787 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:06.449731    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6bf6bb686f-67pvg" podStartSLOduration=1.632517601 podStartE2EDuration="3.449715108s" podCreationTimestamp="2026-05-26 11:27:03 +0000 UTC" firstStartedPulling="2026-05-26 11:27:03.971437161 +0000 UTC m=+156.777275606" lastFinishedPulling="2026-05-26 11:27:05.788634671 +0000 UTC m=+158.594473113" observedRunningTime="2026-05-26 11:27:06.448903376 +0000 UTC m=+159.254741839" watchObservedRunningTime="2026-05-26 11:27:06.449715108 +0000 UTC m=+159.255553573"
May 26 11:27:07.126942 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.126904    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b46fd8c9c-r8qwv"]
May 26 11:27:07.129061 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.129043    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.141765 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.141742    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b46fd8c9c-r8qwv"]
May 26 11:27:07.153703 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.153680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-oauth-serving-cert\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.153792 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.153716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nljs\" (UniqueName: \"kubernetes.io/projected/6193a6d3-2aa6-49ac-857d-98f737ee0d38-kube-api-access-5nljs\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.153792 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.153785    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-trusted-ca-bundle\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.153861 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.153802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-oauth-config\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.153861 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.153825    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-config\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.153861 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.153847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-service-ca\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.153949 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.153869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-serving-cert\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.255087 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.255049    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-trusted-ca-bundle\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.255087 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.255087    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-oauth-config\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.255320 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.255106    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-config\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.255320 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.255237    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-service-ca\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.255401 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.255320    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-serving-cert\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.255401 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.255352    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-oauth-serving-cert\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.255401 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.255395    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5nljs\" (UniqueName: \"kubernetes.io/projected/6193a6d3-2aa6-49ac-857d-98f737ee0d38-kube-api-access-5nljs\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.255991 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.255961    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-service-ca\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.255991 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.255977    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-config\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.256168 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.256039    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-oauth-serving-cert\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.256168 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.256058    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-trusted-ca-bundle\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.257532 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.257512    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-oauth-config\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.257741 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.257725    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-serving-cert\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.263560 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.263536    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nljs\" (UniqueName: \"kubernetes.io/projected/6193a6d3-2aa6-49ac-857d-98f737ee0d38-kube-api-access-5nljs\") pod \"console-5b46fd8c9c-r8qwv\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") " pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.438609 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.438535    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:07.565410 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:07.565348    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b46fd8c9c-r8qwv"]
May 26 11:27:07.567846 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:27:07.567815    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6193a6d3_2aa6_49ac_857d_98f737ee0d38.slice/crio-bb52982cfae2889d185ff120c1babe97bad3aadd01218973b2b8b3d6cb07df66 WatchSource:0}: Error finding container bb52982cfae2889d185ff120c1babe97bad3aadd01218973b2b8b3d6cb07df66: Status 404 returned error can't find the container with id bb52982cfae2889d185ff120c1babe97bad3aadd01218973b2b8b3d6cb07df66
May 26 11:27:08.435516 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:08.435476    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b46fd8c9c-r8qwv" event={"ID":"6193a6d3-2aa6-49ac-857d-98f737ee0d38","Type":"ContainerStarted","Data":"dbadc9a9d25e3c55c3b70a67e8b681bfb0c65d2f40cbd14dbd890eb4686cedfe"}
May 26 11:27:08.435516 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:08.435518    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b46fd8c9c-r8qwv" event={"ID":"6193a6d3-2aa6-49ac-857d-98f737ee0d38","Type":"ContainerStarted","Data":"bb52982cfae2889d185ff120c1babe97bad3aadd01218973b2b8b3d6cb07df66"}
May 26 11:27:08.451974 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:08.451918    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5b46fd8c9c-r8qwv" podStartSLOduration=1.451902862 podStartE2EDuration="1.451902862s" podCreationTimestamp="2026-05-26 11:27:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:27:08.451165873 +0000 UTC m=+161.257004343" watchObservedRunningTime="2026-05-26 11:27:08.451902862 +0000 UTC m=+161.257741328"
May 26 11:27:17.439089 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:17.438995    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:17.439089 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:17.439049    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:17.443830 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:17.443805    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:17.464098 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:17.464074    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:27:17.508306 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:17.508254    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65ddff8745-p5kb2"]
May 26 11:27:42.528185 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.528117    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-65ddff8745-p5kb2" podUID="2909cd2e-db19-4b80-b408-34ecaaf971e9" containerName="console" containerID="cri-o://931f76bc7f2f826882a2d4504999881911fa9f16794147bc014ba0a76582247e" gracePeriod=15
May 26 11:27:42.764261 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.764241    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65ddff8745-p5kb2_2909cd2e-db19-4b80-b408-34ecaaf971e9/console/0.log"
May 26 11:27:42.764396 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.764322    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:27:42.847122 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.847043    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-serving-cert\") pod \"2909cd2e-db19-4b80-b408-34ecaaf971e9\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") "
May 26 11:27:42.847122 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.847075    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-oauth-serving-cert\") pod \"2909cd2e-db19-4b80-b408-34ecaaf971e9\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") "
May 26 11:27:42.847122 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.847112    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-service-ca\") pod \"2909cd2e-db19-4b80-b408-34ecaaf971e9\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") "
May 26 11:27:42.847420 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.847136    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-oauth-config\") pod \"2909cd2e-db19-4b80-b408-34ecaaf971e9\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") "
May 26 11:27:42.847420 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.847171    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-config\") pod \"2909cd2e-db19-4b80-b408-34ecaaf971e9\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") "
May 26 11:27:42.847420 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.847220    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-trusted-ca-bundle\") pod \"2909cd2e-db19-4b80-b408-34ecaaf971e9\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") "
May 26 11:27:42.847420 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.847253    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhh84\" (UniqueName: \"kubernetes.io/projected/2909cd2e-db19-4b80-b408-34ecaaf971e9-kube-api-access-zhh84\") pod \"2909cd2e-db19-4b80-b408-34ecaaf971e9\" (UID: \"2909cd2e-db19-4b80-b408-34ecaaf971e9\") "
May 26 11:27:42.847615 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.847589    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2909cd2e-db19-4b80-b408-34ecaaf971e9" (UID: "2909cd2e-db19-4b80-b408-34ecaaf971e9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:27:42.847662 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.847634    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-service-ca" (OuterVolumeSpecName: "service-ca") pod "2909cd2e-db19-4b80-b408-34ecaaf971e9" (UID: "2909cd2e-db19-4b80-b408-34ecaaf971e9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:27:42.847662 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.847638    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-config" (OuterVolumeSpecName: "console-config") pod "2909cd2e-db19-4b80-b408-34ecaaf971e9" (UID: "2909cd2e-db19-4b80-b408-34ecaaf971e9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:27:42.847773 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.847700    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2909cd2e-db19-4b80-b408-34ecaaf971e9" (UID: "2909cd2e-db19-4b80-b408-34ecaaf971e9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:27:42.849228 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.849206    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2909cd2e-db19-4b80-b408-34ecaaf971e9" (UID: "2909cd2e-db19-4b80-b408-34ecaaf971e9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:42.849316 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.849221    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2909cd2e-db19-4b80-b408-34ecaaf971e9" (UID: "2909cd2e-db19-4b80-b408-34ecaaf971e9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:42.849423 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.849405    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2909cd2e-db19-4b80-b408-34ecaaf971e9-kube-api-access-zhh84" (OuterVolumeSpecName: "kube-api-access-zhh84") pod "2909cd2e-db19-4b80-b408-34ecaaf971e9" (UID: "2909cd2e-db19-4b80-b408-34ecaaf971e9"). InnerVolumeSpecName "kube-api-access-zhh84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:27:42.947971 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.947937    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:42.947971 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.947967    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-trusted-ca-bundle\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:42.948147 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.947982    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhh84\" (UniqueName: \"kubernetes.io/projected/2909cd2e-db19-4b80-b408-34ecaaf971e9-kube-api-access-zhh84\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:42.948147 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.948000    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-serving-cert\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:42.948147 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.948009    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-oauth-serving-cert\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:42.948147 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.948017    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2909cd2e-db19-4b80-b408-34ecaaf971e9-service-ca\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:42.948147 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:42.948025    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2909cd2e-db19-4b80-b408-34ecaaf971e9-console-oauth-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:27:43.535202 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:43.535171    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65ddff8745-p5kb2_2909cd2e-db19-4b80-b408-34ecaaf971e9/console/0.log"
May 26 11:27:43.535620 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:43.535209    2563 generic.go:358] "Generic (PLEG): container finished" podID="2909cd2e-db19-4b80-b408-34ecaaf971e9" containerID="931f76bc7f2f826882a2d4504999881911fa9f16794147bc014ba0a76582247e" exitCode=2
May 26 11:27:43.535620 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:43.535243    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65ddff8745-p5kb2" event={"ID":"2909cd2e-db19-4b80-b408-34ecaaf971e9","Type":"ContainerDied","Data":"931f76bc7f2f826882a2d4504999881911fa9f16794147bc014ba0a76582247e"}
May 26 11:27:43.535620 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:43.535276    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65ddff8745-p5kb2"
May 26 11:27:43.535620 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:43.535300    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65ddff8745-p5kb2" event={"ID":"2909cd2e-db19-4b80-b408-34ecaaf971e9","Type":"ContainerDied","Data":"977319bf2bf79ff5cd920e617d2cf272bf6a53c05d0626aa60038024d6fbf15c"}
May 26 11:27:43.535620 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:43.535321    2563 scope.go:117] "RemoveContainer" containerID="931f76bc7f2f826882a2d4504999881911fa9f16794147bc014ba0a76582247e"
May 26 11:27:43.543813 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:43.543779    2563 scope.go:117] "RemoveContainer" containerID="931f76bc7f2f826882a2d4504999881911fa9f16794147bc014ba0a76582247e"
May 26 11:27:43.544068 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:27:43.544047    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"931f76bc7f2f826882a2d4504999881911fa9f16794147bc014ba0a76582247e\": container with ID starting with 931f76bc7f2f826882a2d4504999881911fa9f16794147bc014ba0a76582247e not found: ID does not exist" containerID="931f76bc7f2f826882a2d4504999881911fa9f16794147bc014ba0a76582247e"
May 26 11:27:43.544138 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:43.544077    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931f76bc7f2f826882a2d4504999881911fa9f16794147bc014ba0a76582247e"} err="failed to get container status \"931f76bc7f2f826882a2d4504999881911fa9f16794147bc014ba0a76582247e\": rpc error: code = NotFound desc = could not find container \"931f76bc7f2f826882a2d4504999881911fa9f16794147bc014ba0a76582247e\": container with ID starting with 931f76bc7f2f826882a2d4504999881911fa9f16794147bc014ba0a76582247e not found: ID does not exist"
May 26 11:27:43.556188 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:43.556162    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65ddff8745-p5kb2"]
May 26 11:27:43.559184 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:43.559163    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65ddff8745-p5kb2"]
May 26 11:27:43.795737 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:27:43.795663    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2909cd2e-db19-4b80-b408-34ecaaf971e9" path="/var/lib/kubelet/pods/2909cd2e-db19-4b80-b408-34ecaaf971e9/volumes"
May 26 11:28:29.055772 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.055738    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fb64dfbf6-cqn8d"]
May 26 11:28:29.056156 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.056077    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2909cd2e-db19-4b80-b408-34ecaaf971e9" containerName="console"
May 26 11:28:29.056156 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.056090    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2909cd2e-db19-4b80-b408-34ecaaf971e9" containerName="console"
May 26 11:28:29.056156 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.056154    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="2909cd2e-db19-4b80-b408-34ecaaf971e9" containerName="console"
May 26 11:28:29.060080 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.060061    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.068880 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.068855    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fb64dfbf6-cqn8d"]
May 26 11:28:29.117161 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.117133    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-service-ca\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.117301 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.117170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d534b47d-921e-4870-955d-7457398866a6-console-serving-cert\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.117301 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.117188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-oauth-serving-cert\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.117385 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.117296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d534b47d-921e-4870-955d-7457398866a6-console-oauth-config\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.117385 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.117325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8m4h\" (UniqueName: \"kubernetes.io/projected/d534b47d-921e-4870-955d-7457398866a6-kube-api-access-v8m4h\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.117385 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.117347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-console-config\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.117385 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.117375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-trusted-ca-bundle\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.218082 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.218050    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d534b47d-921e-4870-955d-7457398866a6-console-oauth-config\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.218082 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.218085    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v8m4h\" (UniqueName: \"kubernetes.io/projected/d534b47d-921e-4870-955d-7457398866a6-kube-api-access-v8m4h\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.218326 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.218106    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-console-config\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.218326 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.218179    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-trusted-ca-bundle\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.218326 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.218247    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-service-ca\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.218326 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.218311    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d534b47d-921e-4870-955d-7457398866a6-console-serving-cert\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.218542 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.218334    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-oauth-serving-cert\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.218871 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.218844    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-console-config\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.218961 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.218944    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-service-ca\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.219054 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.219035    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-trusted-ca-bundle\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.219114 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.219037    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-oauth-serving-cert\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.221200 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.221182    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d534b47d-921e-4870-955d-7457398866a6-console-serving-cert\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.221335 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.221315    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d534b47d-921e-4870-955d-7457398866a6-console-oauth-config\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.228602 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.228583    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8m4h\" (UniqueName: \"kubernetes.io/projected/d534b47d-921e-4870-955d-7457398866a6-kube-api-access-v8m4h\") pod \"console-6fb64dfbf6-cqn8d\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") " pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.368971 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.368893    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:29.514017 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.513977    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fb64dfbf6-cqn8d"]
May 26 11:28:29.516812 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:28:29.516783    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd534b47d_921e_4870_955d_7457398866a6.slice/crio-53558f6a7d14d43a6b448fb500390c32a384fde8c7bc78af0722801264915fdd WatchSource:0}: Error finding container 53558f6a7d14d43a6b448fb500390c32a384fde8c7bc78af0722801264915fdd: Status 404 returned error can't find the container with id 53558f6a7d14d43a6b448fb500390c32a384fde8c7bc78af0722801264915fdd
May 26 11:28:29.666385 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.666307    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb64dfbf6-cqn8d" event={"ID":"d534b47d-921e-4870-955d-7457398866a6","Type":"ContainerStarted","Data":"80bd0021260893ca6efa70bd35f22cd0ed1d5211971222bcaca12a486213c703"}
May 26 11:28:29.666385 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.666344    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb64dfbf6-cqn8d" event={"ID":"d534b47d-921e-4870-955d-7457398866a6","Type":"ContainerStarted","Data":"53558f6a7d14d43a6b448fb500390c32a384fde8c7bc78af0722801264915fdd"}
May 26 11:28:29.683215 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:29.683164    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6fb64dfbf6-cqn8d" podStartSLOduration=0.683150142 podStartE2EDuration="683.150142ms" podCreationTimestamp="2026-05-26 11:28:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:28:29.682397095 +0000 UTC m=+242.488235570" watchObservedRunningTime="2026-05-26 11:28:29.683150142 +0000 UTC m=+242.488988605"
May 26 11:28:39.369612 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:39.369551    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:39.369612 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:39.369608    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:39.374379 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:39.374356    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:39.697480 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:39.697404    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:28:39.779375 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:28:39.779332    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b46fd8c9c-r8qwv"]
May 26 11:29:04.799369 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:04.799225    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5b46fd8c9c-r8qwv" podUID="6193a6d3-2aa6-49ac-857d-98f737ee0d38" containerName="console" containerID="cri-o://dbadc9a9d25e3c55c3b70a67e8b681bfb0c65d2f40cbd14dbd890eb4686cedfe" gracePeriod=15
May 26 11:29:05.038860 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.038836    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b46fd8c9c-r8qwv_6193a6d3-2aa6-49ac-857d-98f737ee0d38/console/0.log"
May 26 11:29:05.038971 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.038903    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:29:05.116904 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.116805    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-trusted-ca-bundle\") pod \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") "
May 26 11:29:05.116904 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.116858    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nljs\" (UniqueName: \"kubernetes.io/projected/6193a6d3-2aa6-49ac-857d-98f737ee0d38-kube-api-access-5nljs\") pod \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") "
May 26 11:29:05.116904 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.116884    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-service-ca\") pod \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") "
May 26 11:29:05.116904 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.116904    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-serving-cert\") pod \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") "
May 26 11:29:05.117225 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.116931    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-oauth-config\") pod \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") "
May 26 11:29:05.117225 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.116946    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-config\") pod \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") "
May 26 11:29:05.117225 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.116973    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-oauth-serving-cert\") pod \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\" (UID: \"6193a6d3-2aa6-49ac-857d-98f737ee0d38\") "
May 26 11:29:05.117437 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.117349    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6193a6d3-2aa6-49ac-857d-98f737ee0d38" (UID: "6193a6d3-2aa6-49ac-857d-98f737ee0d38"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:29:05.117491 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.117440    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-config" (OuterVolumeSpecName: "console-config") pod "6193a6d3-2aa6-49ac-857d-98f737ee0d38" (UID: "6193a6d3-2aa6-49ac-857d-98f737ee0d38"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:29:05.117491 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.117452    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-service-ca" (OuterVolumeSpecName: "service-ca") pod "6193a6d3-2aa6-49ac-857d-98f737ee0d38" (UID: "6193a6d3-2aa6-49ac-857d-98f737ee0d38"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:29:05.117491 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.117480    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6193a6d3-2aa6-49ac-857d-98f737ee0d38" (UID: "6193a6d3-2aa6-49ac-857d-98f737ee0d38"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:29:05.119117 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.119087    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6193a6d3-2aa6-49ac-857d-98f737ee0d38" (UID: "6193a6d3-2aa6-49ac-857d-98f737ee0d38"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:05.119234 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.119130    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6193a6d3-2aa6-49ac-857d-98f737ee0d38" (UID: "6193a6d3-2aa6-49ac-857d-98f737ee0d38"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:05.119234 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.119197    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6193a6d3-2aa6-49ac-857d-98f737ee0d38-kube-api-access-5nljs" (OuterVolumeSpecName: "kube-api-access-5nljs") pod "6193a6d3-2aa6-49ac-857d-98f737ee0d38" (UID: "6193a6d3-2aa6-49ac-857d-98f737ee0d38"). InnerVolumeSpecName "kube-api-access-5nljs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:29:05.218561 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.218524    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-trusted-ca-bundle\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:29:05.218561 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.218553    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nljs\" (UniqueName: \"kubernetes.io/projected/6193a6d3-2aa6-49ac-857d-98f737ee0d38-kube-api-access-5nljs\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:29:05.218561 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.218565    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-service-ca\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:29:05.218794 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.218574    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-serving-cert\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:29:05.218794 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.218583    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-oauth-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:29:05.218794 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.218592    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-console-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:29:05.218794 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.218602    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6193a6d3-2aa6-49ac-857d-98f737ee0d38-oauth-serving-cert\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:29:05.767938 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.767912    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b46fd8c9c-r8qwv_6193a6d3-2aa6-49ac-857d-98f737ee0d38/console/0.log"
May 26 11:29:05.768098 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.767954    2563 generic.go:358] "Generic (PLEG): container finished" podID="6193a6d3-2aa6-49ac-857d-98f737ee0d38" containerID="dbadc9a9d25e3c55c3b70a67e8b681bfb0c65d2f40cbd14dbd890eb4686cedfe" exitCode=2
May 26 11:29:05.768098 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.768001    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b46fd8c9c-r8qwv" event={"ID":"6193a6d3-2aa6-49ac-857d-98f737ee0d38","Type":"ContainerDied","Data":"dbadc9a9d25e3c55c3b70a67e8b681bfb0c65d2f40cbd14dbd890eb4686cedfe"}
May 26 11:29:05.768098 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.768040    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b46fd8c9c-r8qwv"
May 26 11:29:05.768098 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.768047    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b46fd8c9c-r8qwv" event={"ID":"6193a6d3-2aa6-49ac-857d-98f737ee0d38","Type":"ContainerDied","Data":"bb52982cfae2889d185ff120c1babe97bad3aadd01218973b2b8b3d6cb07df66"}
May 26 11:29:05.768098 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.768066    2563 scope.go:117] "RemoveContainer" containerID="dbadc9a9d25e3c55c3b70a67e8b681bfb0c65d2f40cbd14dbd890eb4686cedfe"
May 26 11:29:05.776463 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.776446    2563 scope.go:117] "RemoveContainer" containerID="dbadc9a9d25e3c55c3b70a67e8b681bfb0c65d2f40cbd14dbd890eb4686cedfe"
May 26 11:29:05.776737 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:29:05.776719    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbadc9a9d25e3c55c3b70a67e8b681bfb0c65d2f40cbd14dbd890eb4686cedfe\": container with ID starting with dbadc9a9d25e3c55c3b70a67e8b681bfb0c65d2f40cbd14dbd890eb4686cedfe not found: ID does not exist" containerID="dbadc9a9d25e3c55c3b70a67e8b681bfb0c65d2f40cbd14dbd890eb4686cedfe"
May 26 11:29:05.776814 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.776745    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbadc9a9d25e3c55c3b70a67e8b681bfb0c65d2f40cbd14dbd890eb4686cedfe"} err="failed to get container status \"dbadc9a9d25e3c55c3b70a67e8b681bfb0c65d2f40cbd14dbd890eb4686cedfe\": rpc error: code = NotFound desc = could not find container \"dbadc9a9d25e3c55c3b70a67e8b681bfb0c65d2f40cbd14dbd890eb4686cedfe\": container with ID starting with dbadc9a9d25e3c55c3b70a67e8b681bfb0c65d2f40cbd14dbd890eb4686cedfe not found: ID does not exist"
May 26 11:29:05.788106 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.788085    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b46fd8c9c-r8qwv"]
May 26 11:29:05.794731 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:05.794713    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b46fd8c9c-r8qwv"]
May 26 11:29:07.794881 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:07.794849    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6193a6d3-2aa6-49ac-857d-98f737ee0d38" path="/var/lib/kubelet/pods/6193a6d3-2aa6-49ac-857d-98f737ee0d38/volumes"
May 26 11:29:27.686876 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:29:27.686841    2563 kubelet.go:1628] "Image garbage collection succeeded"
May 26 11:32:08.550389 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.550347    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b56c5bfb5-2btzw"]
May 26 11:32:08.550862 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.550645    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6193a6d3-2aa6-49ac-857d-98f737ee0d38" containerName="console"
May 26 11:32:08.550862 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.550657    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6193a6d3-2aa6-49ac-857d-98f737ee0d38" containerName="console"
May 26 11:32:08.550862 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.550712    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="6193a6d3-2aa6-49ac-857d-98f737ee0d38" containerName="console"
May 26 11:32:08.553515 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.553496    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.586263 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.586229    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b56c5bfb5-2btzw"]
May 26 11:32:08.653220 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.653181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/500d8607-95ab-4f67-889a-66b501c166c7-console-serving-cert\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.653415 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.653230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/500d8607-95ab-4f67-889a-66b501c166c7-console-config\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.653415 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.653255    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/500d8607-95ab-4f67-889a-66b501c166c7-trusted-ca-bundle\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.653415 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.653331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/500d8607-95ab-4f67-889a-66b501c166c7-console-oauth-config\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.653415 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.653362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw942\" (UniqueName: \"kubernetes.io/projected/500d8607-95ab-4f67-889a-66b501c166c7-kube-api-access-qw942\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.653415 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.653392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/500d8607-95ab-4f67-889a-66b501c166c7-service-ca\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.653592 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.653467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/500d8607-95ab-4f67-889a-66b501c166c7-oauth-serving-cert\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.753943 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.753911    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/500d8607-95ab-4f67-889a-66b501c166c7-console-serving-cert\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.753943 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.753954    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/500d8607-95ab-4f67-889a-66b501c166c7-console-config\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.754217 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.753974    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/500d8607-95ab-4f67-889a-66b501c166c7-trusted-ca-bundle\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.754217 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.754031    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/500d8607-95ab-4f67-889a-66b501c166c7-console-oauth-config\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.754217 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.754065    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qw942\" (UniqueName: \"kubernetes.io/projected/500d8607-95ab-4f67-889a-66b501c166c7-kube-api-access-qw942\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.754217 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.754096    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/500d8607-95ab-4f67-889a-66b501c166c7-service-ca\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.754217 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.754121    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/500d8607-95ab-4f67-889a-66b501c166c7-oauth-serving-cert\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.754889 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.754864    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/500d8607-95ab-4f67-889a-66b501c166c7-console-config\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.755000 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.754909    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/500d8607-95ab-4f67-889a-66b501c166c7-oauth-serving-cert\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.755000 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.754993    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/500d8607-95ab-4f67-889a-66b501c166c7-trusted-ca-bundle\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.755110 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.754994    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/500d8607-95ab-4f67-889a-66b501c166c7-service-ca\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.756547 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.756521    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/500d8607-95ab-4f67-889a-66b501c166c7-console-serving-cert\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.756645 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.756562    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/500d8607-95ab-4f67-889a-66b501c166c7-console-oauth-config\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.762502 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.762473    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw942\" (UniqueName: \"kubernetes.io/projected/500d8607-95ab-4f67-889a-66b501c166c7-kube-api-access-qw942\") pod \"console-5b56c5bfb5-2btzw\" (UID: \"500d8607-95ab-4f67-889a-66b501c166c7\") " pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.861841 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.861762    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:08.984221 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.984192    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b56c5bfb5-2btzw"]
May 26 11:32:08.987255 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:32:08.987219    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500d8607_95ab_4f67_889a_66b501c166c7.slice/crio-61aab5a4e926c5f9c7c5af4a9d63c939e78f96030f76b9d61890b18127b6cf7f WatchSource:0}: Error finding container 61aab5a4e926c5f9c7c5af4a9d63c939e78f96030f76b9d61890b18127b6cf7f: Status 404 returned error can't find the container with id 61aab5a4e926c5f9c7c5af4a9d63c939e78f96030f76b9d61890b18127b6cf7f
May 26 11:32:08.989163 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:08.989146    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:32:09.266312 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:09.266266    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b56c5bfb5-2btzw" event={"ID":"500d8607-95ab-4f67-889a-66b501c166c7","Type":"ContainerStarted","Data":"d01db1b63d8b054bcf1aa2d547b7c31a85d733b2674b8f84b2be876d54806164"}
May 26 11:32:09.266312 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:09.266316    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b56c5bfb5-2btzw" event={"ID":"500d8607-95ab-4f67-889a-66b501c166c7","Type":"ContainerStarted","Data":"61aab5a4e926c5f9c7c5af4a9d63c939e78f96030f76b9d61890b18127b6cf7f"}
May 26 11:32:09.282814 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:09.282776    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5b56c5bfb5-2btzw" podStartSLOduration=1.282763143 podStartE2EDuration="1.282763143s" podCreationTimestamp="2026-05-26 11:32:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:32:09.281822615 +0000 UTC m=+462.087661079" watchObservedRunningTime="2026-05-26 11:32:09.282763143 +0000 UTC m=+462.088601604"
May 26 11:32:18.862782 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:18.862706    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:18.862782 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:18.862747    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:18.867270 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:18.867248    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:19.296832 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:19.296800    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b56c5bfb5-2btzw"
May 26 11:32:19.346321 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:19.346267    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fb64dfbf6-cqn8d"]
May 26 11:32:44.365372 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.365331    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-6fb64dfbf6-cqn8d" podUID="d534b47d-921e-4870-955d-7457398866a6" containerName="console" containerID="cri-o://80bd0021260893ca6efa70bd35f22cd0ed1d5211971222bcaca12a486213c703" gracePeriod=15
May 26 11:32:44.596148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.596126    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6fb64dfbf6-cqn8d_d534b47d-921e-4870-955d-7457398866a6/console/0.log"
May 26 11:32:44.596273 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.596189    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:32:44.653271 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.653202    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-console-config\") pod \"d534b47d-921e-4870-955d-7457398866a6\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") "
May 26 11:32:44.653271 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.653235    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8m4h\" (UniqueName: \"kubernetes.io/projected/d534b47d-921e-4870-955d-7457398866a6-kube-api-access-v8m4h\") pod \"d534b47d-921e-4870-955d-7457398866a6\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") "
May 26 11:32:44.653271 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.653260    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d534b47d-921e-4870-955d-7457398866a6-console-serving-cert\") pod \"d534b47d-921e-4870-955d-7457398866a6\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") "
May 26 11:32:44.653512 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.653278    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-trusted-ca-bundle\") pod \"d534b47d-921e-4870-955d-7457398866a6\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") "
May 26 11:32:44.653512 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.653330    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d534b47d-921e-4870-955d-7457398866a6-console-oauth-config\") pod \"d534b47d-921e-4870-955d-7457398866a6\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") "
May 26 11:32:44.653512 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.653374    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-service-ca\") pod \"d534b47d-921e-4870-955d-7457398866a6\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") "
May 26 11:32:44.653512 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.653399    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-oauth-serving-cert\") pod \"d534b47d-921e-4870-955d-7457398866a6\" (UID: \"d534b47d-921e-4870-955d-7457398866a6\") "
May 26 11:32:44.653830 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.653785    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-console-config" (OuterVolumeSpecName: "console-config") pod "d534b47d-921e-4870-955d-7457398866a6" (UID: "d534b47d-921e-4870-955d-7457398866a6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:32:44.653944 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.653828    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d534b47d-921e-4870-955d-7457398866a6" (UID: "d534b47d-921e-4870-955d-7457398866a6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:32:44.653944 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.653907    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d534b47d-921e-4870-955d-7457398866a6" (UID: "d534b47d-921e-4870-955d-7457398866a6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:32:44.653944 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.653928    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-service-ca" (OuterVolumeSpecName: "service-ca") pod "d534b47d-921e-4870-955d-7457398866a6" (UID: "d534b47d-921e-4870-955d-7457398866a6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:32:44.655466 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.655438    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d534b47d-921e-4870-955d-7457398866a6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d534b47d-921e-4870-955d-7457398866a6" (UID: "d534b47d-921e-4870-955d-7457398866a6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:32:44.655466 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.655454    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d534b47d-921e-4870-955d-7457398866a6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d534b47d-921e-4870-955d-7457398866a6" (UID: "d534b47d-921e-4870-955d-7457398866a6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:32:44.655577 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.655512    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d534b47d-921e-4870-955d-7457398866a6-kube-api-access-v8m4h" (OuterVolumeSpecName: "kube-api-access-v8m4h") pod "d534b47d-921e-4870-955d-7457398866a6" (UID: "d534b47d-921e-4870-955d-7457398866a6"). InnerVolumeSpecName "kube-api-access-v8m4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:32:44.754761 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.754729    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-console-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:32:44.754761 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.754754    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8m4h\" (UniqueName: \"kubernetes.io/projected/d534b47d-921e-4870-955d-7457398866a6-kube-api-access-v8m4h\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:32:44.754956 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.754778    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d534b47d-921e-4870-955d-7457398866a6-console-serving-cert\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:32:44.754956 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.754787    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-trusted-ca-bundle\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:32:44.754956 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.754796    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d534b47d-921e-4870-955d-7457398866a6-console-oauth-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:32:44.754956 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.754805    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-service-ca\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:32:44.754956 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:44.754814    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d534b47d-921e-4870-955d-7457398866a6-oauth-serving-cert\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:32:45.372953 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:45.372926    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6fb64dfbf6-cqn8d_d534b47d-921e-4870-955d-7457398866a6/console/0.log"
May 26 11:32:45.373347 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:45.372964    2563 generic.go:358] "Generic (PLEG): container finished" podID="d534b47d-921e-4870-955d-7457398866a6" containerID="80bd0021260893ca6efa70bd35f22cd0ed1d5211971222bcaca12a486213c703" exitCode=2
May 26 11:32:45.373347 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:45.372997    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb64dfbf6-cqn8d" event={"ID":"d534b47d-921e-4870-955d-7457398866a6","Type":"ContainerDied","Data":"80bd0021260893ca6efa70bd35f22cd0ed1d5211971222bcaca12a486213c703"}
May 26 11:32:45.373347 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:45.373043    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb64dfbf6-cqn8d" event={"ID":"d534b47d-921e-4870-955d-7457398866a6","Type":"ContainerDied","Data":"53558f6a7d14d43a6b448fb500390c32a384fde8c7bc78af0722801264915fdd"}
May 26 11:32:45.373347 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:45.373063    2563 scope.go:117] "RemoveContainer" containerID="80bd0021260893ca6efa70bd35f22cd0ed1d5211971222bcaca12a486213c703"
May 26 11:32:45.373347 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:45.373065    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb64dfbf6-cqn8d"
May 26 11:32:45.381100 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:45.381085    2563 scope.go:117] "RemoveContainer" containerID="80bd0021260893ca6efa70bd35f22cd0ed1d5211971222bcaca12a486213c703"
May 26 11:32:45.381343 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:32:45.381326    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80bd0021260893ca6efa70bd35f22cd0ed1d5211971222bcaca12a486213c703\": container with ID starting with 80bd0021260893ca6efa70bd35f22cd0ed1d5211971222bcaca12a486213c703 not found: ID does not exist" containerID="80bd0021260893ca6efa70bd35f22cd0ed1d5211971222bcaca12a486213c703"
May 26 11:32:45.381415 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:45.381349    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80bd0021260893ca6efa70bd35f22cd0ed1d5211971222bcaca12a486213c703"} err="failed to get container status \"80bd0021260893ca6efa70bd35f22cd0ed1d5211971222bcaca12a486213c703\": rpc error: code = NotFound desc = could not find container \"80bd0021260893ca6efa70bd35f22cd0ed1d5211971222bcaca12a486213c703\": container with ID starting with 80bd0021260893ca6efa70bd35f22cd0ed1d5211971222bcaca12a486213c703 not found: ID does not exist"
May 26 11:32:45.395183 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:45.395161    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fb64dfbf6-cqn8d"]
May 26 11:32:45.398190 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:45.398167    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fb64dfbf6-cqn8d"]
May 26 11:32:45.795645 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:32:45.795610    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d534b47d-921e-4870-955d-7457398866a6" path="/var/lib/kubelet/pods/d534b47d-921e-4870-955d-7457398866a6/volumes"
May 26 11:39:35.787360 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.787316    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl"]
May 26 11:39:35.787793 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.787622    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d534b47d-921e-4870-955d-7457398866a6" containerName="console"
May 26 11:39:35.787793 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.787633    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d534b47d-921e-4870-955d-7457398866a6" containerName="console"
May 26 11:39:35.787793 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.787696    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="d534b47d-921e-4870-955d-7457398866a6" containerName="console"
May 26 11:39:35.790670 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.790653    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl"
May 26 11:39:35.793685 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.793653    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 26 11:39:35.794528 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.794503    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kddrj\""
May 26 11:39:35.795958 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.795934    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 26 11:39:35.811726 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.811702    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl"]
May 26 11:39:35.874498 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.874457    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf4b54b6-03f8-428e-b31b-d83ac1515075-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl\" (UID: \"cf4b54b6-03f8-428e-b31b-d83ac1515075\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl"
May 26 11:39:35.874673 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.874508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr84j\" (UniqueName: \"kubernetes.io/projected/cf4b54b6-03f8-428e-b31b-d83ac1515075-kube-api-access-hr84j\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl\" (UID: \"cf4b54b6-03f8-428e-b31b-d83ac1515075\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl"
May 26 11:39:35.874673 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.874609    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf4b54b6-03f8-428e-b31b-d83ac1515075-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl\" (UID: \"cf4b54b6-03f8-428e-b31b-d83ac1515075\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl"
May 26 11:39:35.975389 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.975353    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf4b54b6-03f8-428e-b31b-d83ac1515075-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl\" (UID: \"cf4b54b6-03f8-428e-b31b-d83ac1515075\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl"
May 26 11:39:35.975548 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.975402    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hr84j\" (UniqueName: \"kubernetes.io/projected/cf4b54b6-03f8-428e-b31b-d83ac1515075-kube-api-access-hr84j\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl\" (UID: \"cf4b54b6-03f8-428e-b31b-d83ac1515075\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl"
May 26 11:39:35.975548 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.975436    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf4b54b6-03f8-428e-b31b-d83ac1515075-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl\" (UID: \"cf4b54b6-03f8-428e-b31b-d83ac1515075\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl"
May 26 11:39:35.975761 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.975739    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf4b54b6-03f8-428e-b31b-d83ac1515075-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl\" (UID: \"cf4b54b6-03f8-428e-b31b-d83ac1515075\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl"
May 26 11:39:35.975761 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.975758    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf4b54b6-03f8-428e-b31b-d83ac1515075-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl\" (UID: \"cf4b54b6-03f8-428e-b31b-d83ac1515075\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl"
May 26 11:39:35.985512 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:35.985482    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr84j\" (UniqueName: \"kubernetes.io/projected/cf4b54b6-03f8-428e-b31b-d83ac1515075-kube-api-access-hr84j\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl\" (UID: \"cf4b54b6-03f8-428e-b31b-d83ac1515075\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl"
May 26 11:39:36.104692 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:36.104582    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl"
May 26 11:39:36.231410 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:36.231320    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl"]
May 26 11:39:36.233683 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:39:36.233654    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4b54b6_03f8_428e_b31b_d83ac1515075.slice/crio-e330be7c271ecd4180a19367c2f04c2d4970151a4cdf0f01ba8e00e2da556220 WatchSource:0}: Error finding container e330be7c271ecd4180a19367c2f04c2d4970151a4cdf0f01ba8e00e2da556220: Status 404 returned error can't find the container with id e330be7c271ecd4180a19367c2f04c2d4970151a4cdf0f01ba8e00e2da556220
May 26 11:39:36.235432 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:36.235413    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:39:36.496564 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:36.496531    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl" event={"ID":"cf4b54b6-03f8-428e-b31b-d83ac1515075","Type":"ContainerStarted","Data":"e330be7c271ecd4180a19367c2f04c2d4970151a4cdf0f01ba8e00e2da556220"}
May 26 11:39:42.515118 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:42.515083    2563 generic.go:358] "Generic (PLEG): container finished" podID="cf4b54b6-03f8-428e-b31b-d83ac1515075" containerID="f09c2177db8ff96ce26d85ad24e7fb71ce9cd5d1ff5fe45400152f0d18ea922f" exitCode=0
May 26 11:39:42.515533 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:42.515172    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl" event={"ID":"cf4b54b6-03f8-428e-b31b-d83ac1515075","Type":"ContainerDied","Data":"f09c2177db8ff96ce26d85ad24e7fb71ce9cd5d1ff5fe45400152f0d18ea922f"}
May 26 11:39:44.522830 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:44.522738    2563 generic.go:358] "Generic (PLEG): container finished" podID="cf4b54b6-03f8-428e-b31b-d83ac1515075" containerID="9c82f587e119d207d4aa6323c64f8775f58733c0c9fd35f4505f87c765ce00e0" exitCode=0
May 26 11:39:44.522830 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:44.522784    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl" event={"ID":"cf4b54b6-03f8-428e-b31b-d83ac1515075","Type":"ContainerDied","Data":"9c82f587e119d207d4aa6323c64f8775f58733c0c9fd35f4505f87c765ce00e0"}
May 26 11:39:50.549902 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:50.549868    2563 generic.go:358] "Generic (PLEG): container finished" podID="cf4b54b6-03f8-428e-b31b-d83ac1515075" containerID="7ff715075ff01d7362e62bbdc347976b2d3b57744d66edd5b0d1ffa408049c1e" exitCode=0
May 26 11:39:50.550358 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:50.549954    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl" event={"ID":"cf4b54b6-03f8-428e-b31b-d83ac1515075","Type":"ContainerDied","Data":"7ff715075ff01d7362e62bbdc347976b2d3b57744d66edd5b0d1ffa408049c1e"}
May 26 11:39:51.675871 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:51.675847    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl"
May 26 11:39:51.814474 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:51.814400    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf4b54b6-03f8-428e-b31b-d83ac1515075-bundle\") pod \"cf4b54b6-03f8-428e-b31b-d83ac1515075\" (UID: \"cf4b54b6-03f8-428e-b31b-d83ac1515075\") "
May 26 11:39:51.814474 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:51.814443    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf4b54b6-03f8-428e-b31b-d83ac1515075-util\") pod \"cf4b54b6-03f8-428e-b31b-d83ac1515075\" (UID: \"cf4b54b6-03f8-428e-b31b-d83ac1515075\") "
May 26 11:39:51.814639 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:51.814482    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr84j\" (UniqueName: \"kubernetes.io/projected/cf4b54b6-03f8-428e-b31b-d83ac1515075-kube-api-access-hr84j\") pod \"cf4b54b6-03f8-428e-b31b-d83ac1515075\" (UID: \"cf4b54b6-03f8-428e-b31b-d83ac1515075\") "
May 26 11:39:51.814961 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:51.814935    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4b54b6-03f8-428e-b31b-d83ac1515075-bundle" (OuterVolumeSpecName: "bundle") pod "cf4b54b6-03f8-428e-b31b-d83ac1515075" (UID: "cf4b54b6-03f8-428e-b31b-d83ac1515075"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:39:51.816707 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:51.816682    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4b54b6-03f8-428e-b31b-d83ac1515075-kube-api-access-hr84j" (OuterVolumeSpecName: "kube-api-access-hr84j") pod "cf4b54b6-03f8-428e-b31b-d83ac1515075" (UID: "cf4b54b6-03f8-428e-b31b-d83ac1515075"). InnerVolumeSpecName "kube-api-access-hr84j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:39:51.819573 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:51.819552    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4b54b6-03f8-428e-b31b-d83ac1515075-util" (OuterVolumeSpecName: "util") pod "cf4b54b6-03f8-428e-b31b-d83ac1515075" (UID: "cf4b54b6-03f8-428e-b31b-d83ac1515075"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:39:51.915766 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:51.915713    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf4b54b6-03f8-428e-b31b-d83ac1515075-bundle\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:39:51.915766 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:51.915758    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf4b54b6-03f8-428e-b31b-d83ac1515075-util\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:39:51.915766 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:51.915771    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr84j\" (UniqueName: \"kubernetes.io/projected/cf4b54b6-03f8-428e-b31b-d83ac1515075-kube-api-access-hr84j\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:39:52.557901 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:52.557868    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl" event={"ID":"cf4b54b6-03f8-428e-b31b-d83ac1515075","Type":"ContainerDied","Data":"e330be7c271ecd4180a19367c2f04c2d4970151a4cdf0f01ba8e00e2da556220"}
May 26 11:39:52.557901 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:52.557904    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e330be7c271ecd4180a19367c2f04c2d4970151a4cdf0f01ba8e00e2da556220"
May 26 11:39:52.558140 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:52.557882    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjxcsl"
May 26 11:39:57.737906 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.737871    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fzddk"]
May 26 11:39:57.738268 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.738166    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf4b54b6-03f8-428e-b31b-d83ac1515075" containerName="pull"
May 26 11:39:57.738268 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.738177    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf4b54b6-03f8-428e-b31b-d83ac1515075" containerName="pull"
May 26 11:39:57.738268 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.738195    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf4b54b6-03f8-428e-b31b-d83ac1515075" containerName="extract"
May 26 11:39:57.738268 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.738201    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf4b54b6-03f8-428e-b31b-d83ac1515075" containerName="extract"
May 26 11:39:57.738268 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.738208    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf4b54b6-03f8-428e-b31b-d83ac1515075" containerName="util"
May 26 11:39:57.738268 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.738213    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf4b54b6-03f8-428e-b31b-d83ac1515075" containerName="util"
May 26 11:39:57.738268 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.738257    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="cf4b54b6-03f8-428e-b31b-d83ac1515075" containerName="extract"
May 26 11:39:57.740274 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.740257    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fzddk"
May 26 11:39:57.743021 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.742999    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 26 11:39:57.743154 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.743083    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 26 11:39:57.743224 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.743202    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-vtj78\""
May 26 11:39:57.743406 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.743387    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 26 11:39:57.757838 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.757818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5d421538-81d4-4d1a-ab08-bae247b9f843-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-fzddk\" (UID: \"5d421538-81d4-4d1a-ab08-bae247b9f843\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fzddk"
May 26 11:39:57.757945 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.757869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5j4d\" (UniqueName: \"kubernetes.io/projected/5d421538-81d4-4d1a-ab08-bae247b9f843-kube-api-access-c5j4d\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-fzddk\" (UID: \"5d421538-81d4-4d1a-ab08-bae247b9f843\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fzddk"
May 26 11:39:57.761517 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.761496    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fzddk"]
May 26 11:39:57.858689 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.858658    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c5j4d\" (UniqueName: \"kubernetes.io/projected/5d421538-81d4-4d1a-ab08-bae247b9f843-kube-api-access-c5j4d\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-fzddk\" (UID: \"5d421538-81d4-4d1a-ab08-bae247b9f843\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fzddk"
May 26 11:39:57.858840 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.858711    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5d421538-81d4-4d1a-ab08-bae247b9f843-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-fzddk\" (UID: \"5d421538-81d4-4d1a-ab08-bae247b9f843\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fzddk"
May 26 11:39:57.860877 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.860857    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5d421538-81d4-4d1a-ab08-bae247b9f843-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-fzddk\" (UID: \"5d421538-81d4-4d1a-ab08-bae247b9f843\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fzddk"
May 26 11:39:57.868362 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:57.868341    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5j4d\" (UniqueName: \"kubernetes.io/projected/5d421538-81d4-4d1a-ab08-bae247b9f843-kube-api-access-c5j4d\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-fzddk\" (UID: \"5d421538-81d4-4d1a-ab08-bae247b9f843\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fzddk"
May 26 11:39:58.050744 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:58.050662    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fzddk"
May 26 11:39:58.183007 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:58.182977    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fzddk"]
May 26 11:39:58.187178 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:39:58.187148    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d421538_81d4_4d1a_ab08_bae247b9f843.slice/crio-d8f8ea385205eff89e3f411f1bbb67137983bba8d92db3022d351ac83b38b4b7 WatchSource:0}: Error finding container d8f8ea385205eff89e3f411f1bbb67137983bba8d92db3022d351ac83b38b4b7: Status 404 returned error can't find the container with id d8f8ea385205eff89e3f411f1bbb67137983bba8d92db3022d351ac83b38b4b7
May 26 11:39:58.575325 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:39:58.575272    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fzddk" event={"ID":"5d421538-81d4-4d1a-ab08-bae247b9f843","Type":"ContainerStarted","Data":"d8f8ea385205eff89e3f411f1bbb67137983bba8d92db3022d351ac83b38b4b7"}
May 26 11:40:05.309793 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.309754    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-f7wjz"]
May 26 11:40:05.312402 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.312382    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-f7wjz"
May 26 11:40:05.315167 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.315019    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 26 11:40:05.315167 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.315040    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
May 26 11:40:05.315167 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.315137    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-tn2p4\""
May 26 11:40:05.316137 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.316114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75-certificates\") pod \"keda-operator-ffbb595cb-f7wjz\" (UID: \"8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75\") " pod="openshift-keda/keda-operator-ffbb595cb-f7wjz"
May 26 11:40:05.316246 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.316146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75-cabundle0\") pod \"keda-operator-ffbb595cb-f7wjz\" (UID: \"8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75\") " pod="openshift-keda/keda-operator-ffbb595cb-f7wjz"
May 26 11:40:05.316246 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.316228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9529\" (UniqueName: \"kubernetes.io/projected/8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75-kube-api-access-c9529\") pod \"keda-operator-ffbb595cb-f7wjz\" (UID: \"8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75\") " pod="openshift-keda/keda-operator-ffbb595cb-f7wjz"
May 26 11:40:05.323145 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.323118    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-f7wjz"]
May 26 11:40:05.417349 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.417314    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c9529\" (UniqueName: \"kubernetes.io/projected/8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75-kube-api-access-c9529\") pod \"keda-operator-ffbb595cb-f7wjz\" (UID: \"8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75\") " pod="openshift-keda/keda-operator-ffbb595cb-f7wjz"
May 26 11:40:05.417533 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.417383    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75-certificates\") pod \"keda-operator-ffbb595cb-f7wjz\" (UID: \"8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75\") " pod="openshift-keda/keda-operator-ffbb595cb-f7wjz"
May 26 11:40:05.417533 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.417407    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75-cabundle0\") pod \"keda-operator-ffbb595cb-f7wjz\" (UID: \"8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75\") " pod="openshift-keda/keda-operator-ffbb595cb-f7wjz"
May 26 11:40:05.417533 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:05.417529    2563 projected.go:264] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
May 26 11:40:05.417658 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:05.417546    2563 secret.go:281] references non-existent secret key: ca.crt
May 26 11:40:05.417658 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:05.417552    2563 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 26 11:40:05.417658 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:05.417565    2563 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-f7wjz: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
May 26 11:40:05.417658 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:05.417621    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75-certificates podName:8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75 nodeName:}" failed. No retries permitted until 2026-05-26 11:40:05.917606435 +0000 UTC m=+938.723444884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75-certificates") pod "keda-operator-ffbb595cb-f7wjz" (UID: "8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
May 26 11:40:05.418178 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.418156    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75-cabundle0\") pod \"keda-operator-ffbb595cb-f7wjz\" (UID: \"8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75\") " pod="openshift-keda/keda-operator-ffbb595cb-f7wjz"
May 26 11:40:05.427711 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.427687    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9529\" (UniqueName: \"kubernetes.io/projected/8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75-kube-api-access-c9529\") pod \"keda-operator-ffbb595cb-f7wjz\" (UID: \"8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75\") " pod="openshift-keda/keda-operator-ffbb595cb-f7wjz"
May 26 11:40:05.597315 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.597204    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fzddk" event={"ID":"5d421538-81d4-4d1a-ab08-bae247b9f843","Type":"ContainerStarted","Data":"83633f478181beb2e7c37a9a3b799917b3a296ba676f1cb48041fca2bf4448f4"}
May 26 11:40:05.597477 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.597345    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fzddk"
May 26 11:40:05.625747 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.625686    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fzddk" podStartSLOduration=1.967350497 podStartE2EDuration="8.625667662s" podCreationTimestamp="2026-05-26 11:39:57 +0000 UTC" firstStartedPulling="2026-05-26 11:39:58.188687938 +0000 UTC m=+930.994526380" lastFinishedPulling="2026-05-26 11:40:04.847005101 +0000 UTC m=+937.652843545" observedRunningTime="2026-05-26 11:40:05.624101362 +0000 UTC m=+938.429939826" watchObservedRunningTime="2026-05-26 11:40:05.625667662 +0000 UTC m=+938.431506127"
May 26 11:40:05.706860 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.706823    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"]
May 26 11:40:05.709247 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.709230    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"
May 26 11:40:05.711809 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.711789    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
May 26 11:40:05.720190 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.720164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6nfq\" (UniqueName: \"kubernetes.io/projected/be4489bb-28af-4904-ac19-f491dde48310-kube-api-access-p6nfq\") pod \"keda-metrics-apiserver-7c9f485588-pxkg7\" (UID: \"be4489bb-28af-4904-ac19-f491dde48310\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"
May 26 11:40:05.720278 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.720250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/be4489bb-28af-4904-ac19-f491dde48310-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pxkg7\" (UID: \"be4489bb-28af-4904-ac19-f491dde48310\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"
May 26 11:40:05.720342 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.720296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/be4489bb-28af-4904-ac19-f491dde48310-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-pxkg7\" (UID: \"be4489bb-28af-4904-ac19-f491dde48310\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"
May 26 11:40:05.729138 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.729114    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"]
May 26 11:40:05.820786 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.820752    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p6nfq\" (UniqueName: \"kubernetes.io/projected/be4489bb-28af-4904-ac19-f491dde48310-kube-api-access-p6nfq\") pod \"keda-metrics-apiserver-7c9f485588-pxkg7\" (UID: \"be4489bb-28af-4904-ac19-f491dde48310\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"
May 26 11:40:05.820948 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.820819    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/be4489bb-28af-4904-ac19-f491dde48310-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pxkg7\" (UID: \"be4489bb-28af-4904-ac19-f491dde48310\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"
May 26 11:40:05.820948 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.820855    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/be4489bb-28af-4904-ac19-f491dde48310-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-pxkg7\" (UID: \"be4489bb-28af-4904-ac19-f491dde48310\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"
May 26 11:40:05.821062 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:05.820964    2563 secret.go:281] references non-existent secret key: tls.crt
May 26 11:40:05.821062 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:05.820981    2563 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 26 11:40:05.821062 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:05.820998    2563 projected.go:264] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
May 26 11:40:05.821062 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:05.821016    2563 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
May 26 11:40:05.821193 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:05.821079    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be4489bb-28af-4904-ac19-f491dde48310-certificates podName:be4489bb-28af-4904-ac19-f491dde48310 nodeName:}" failed. No retries permitted until 2026-05-26 11:40:06.321060631 +0000 UTC m=+939.126899079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/be4489bb-28af-4904-ac19-f491dde48310-certificates") pod "keda-metrics-apiserver-7c9f485588-pxkg7" (UID: "be4489bb-28af-4904-ac19-f491dde48310") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
May 26 11:40:05.821357 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.821339    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/be4489bb-28af-4904-ac19-f491dde48310-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-pxkg7\" (UID: \"be4489bb-28af-4904-ac19-f491dde48310\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"
May 26 11:40:05.830546 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.830515    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6nfq\" (UniqueName: \"kubernetes.io/projected/be4489bb-28af-4904-ac19-f491dde48310-kube-api-access-p6nfq\") pod \"keda-metrics-apiserver-7c9f485588-pxkg7\" (UID: \"be4489bb-28af-4904-ac19-f491dde48310\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"
May 26 11:40:05.921929 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:05.921833    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75-certificates\") pod \"keda-operator-ffbb595cb-f7wjz\" (UID: \"8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75\") " pod="openshift-keda/keda-operator-ffbb595cb-f7wjz"
May 26 11:40:05.922107 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:05.921994    2563 secret.go:281] references non-existent secret key: ca.crt
May 26 11:40:05.922107 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:05.922014    2563 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 26 11:40:05.922107 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:05.922026    2563 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-f7wjz: references non-existent secret key: ca.crt
May 26 11:40:05.922107 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:05.922094    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75-certificates podName:8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75 nodeName:}" failed. No retries permitted until 2026-05-26 11:40:06.922072851 +0000 UTC m=+939.727911294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75-certificates") pod "keda-operator-ffbb595cb-f7wjz" (UID: "8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75") : references non-existent secret key: ca.crt
May 26 11:40:06.006606 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:06.006570    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-6fhjs"]
May 26 11:40:06.008976 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:06.008957    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-6fhjs"
May 26 11:40:06.011490 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:06.011472    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
May 26 11:40:06.022581 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:06.022553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k97t\" (UniqueName: \"kubernetes.io/projected/20be1137-924b-435d-a28f-4877cc68a105-kube-api-access-6k97t\") pod \"keda-admission-cf49989db-6fhjs\" (UID: \"20be1137-924b-435d-a28f-4877cc68a105\") " pod="openshift-keda/keda-admission-cf49989db-6fhjs"
May 26 11:40:06.022697 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:06.022611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/20be1137-924b-435d-a28f-4877cc68a105-certificates\") pod \"keda-admission-cf49989db-6fhjs\" (UID: \"20be1137-924b-435d-a28f-4877cc68a105\") " pod="openshift-keda/keda-admission-cf49989db-6fhjs"
May 26 11:40:06.027979 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:06.027953    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-6fhjs"]
May 26 11:40:06.123835 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:06.123803    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6k97t\" (UniqueName: \"kubernetes.io/projected/20be1137-924b-435d-a28f-4877cc68a105-kube-api-access-6k97t\") pod \"keda-admission-cf49989db-6fhjs\" (UID: \"20be1137-924b-435d-a28f-4877cc68a105\") " pod="openshift-keda/keda-admission-cf49989db-6fhjs"
May 26 11:40:06.124011 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:06.123859    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/20be1137-924b-435d-a28f-4877cc68a105-certificates\") pod \"keda-admission-cf49989db-6fhjs\" (UID: \"20be1137-924b-435d-a28f-4877cc68a105\") " pod="openshift-keda/keda-admission-cf49989db-6fhjs"
May 26 11:40:06.126442 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:06.126409    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/20be1137-924b-435d-a28f-4877cc68a105-certificates\") pod \"keda-admission-cf49989db-6fhjs\" (UID: \"20be1137-924b-435d-a28f-4877cc68a105\") " pod="openshift-keda/keda-admission-cf49989db-6fhjs"
May 26 11:40:06.132798 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:06.132770    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k97t\" (UniqueName: \"kubernetes.io/projected/20be1137-924b-435d-a28f-4877cc68a105-kube-api-access-6k97t\") pod \"keda-admission-cf49989db-6fhjs\" (UID: \"20be1137-924b-435d-a28f-4877cc68a105\") " pod="openshift-keda/keda-admission-cf49989db-6fhjs"
May 26 11:40:06.319975 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:06.319931    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-6fhjs"
May 26 11:40:06.326202 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:06.326176    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/be4489bb-28af-4904-ac19-f491dde48310-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pxkg7\" (UID: \"be4489bb-28af-4904-ac19-f491dde48310\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"
May 26 11:40:06.326314 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:06.326277    2563 secret.go:281] references non-existent secret key: tls.crt
May 26 11:40:06.326314 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:06.326307    2563 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 26 11:40:06.326400 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:06.326331    2563 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7: references non-existent secret key: tls.crt
May 26 11:40:06.326400 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:40:06.326384    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be4489bb-28af-4904-ac19-f491dde48310-certificates podName:be4489bb-28af-4904-ac19-f491dde48310 nodeName:}" failed. No retries permitted until 2026-05-26 11:40:07.326370163 +0000 UTC m=+940.132208604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/be4489bb-28af-4904-ac19-f491dde48310-certificates") pod "keda-metrics-apiserver-7c9f485588-pxkg7" (UID: "be4489bb-28af-4904-ac19-f491dde48310") : references non-existent secret key: tls.crt
May 26 11:40:06.455181 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:06.455148    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-6fhjs"]
May 26 11:40:06.458223 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:40:06.458190    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20be1137_924b_435d_a28f_4877cc68a105.slice/crio-58e293d9af8f53198fe6e2137e75a10a81673ff76d3b2a902c4400dcf4d09b3b WatchSource:0}: Error finding container 58e293d9af8f53198fe6e2137e75a10a81673ff76d3b2a902c4400dcf4d09b3b: Status 404 returned error can't find the container with id 58e293d9af8f53198fe6e2137e75a10a81673ff76d3b2a902c4400dcf4d09b3b
May 26 11:40:06.601359 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:06.601239    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-6fhjs" event={"ID":"20be1137-924b-435d-a28f-4877cc68a105","Type":"ContainerStarted","Data":"58e293d9af8f53198fe6e2137e75a10a81673ff76d3b2a902c4400dcf4d09b3b"}
May 26 11:40:06.930668 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:06.930569    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75-certificates\") pod \"keda-operator-ffbb595cb-f7wjz\" (UID: \"8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75\") " pod="openshift-keda/keda-operator-ffbb595cb-f7wjz"
May 26 11:40:06.933096 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:06.933072    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75-certificates\") pod \"keda-operator-ffbb595cb-f7wjz\" (UID: \"8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75\") " pod="openshift-keda/keda-operator-ffbb595cb-f7wjz"
May 26 11:40:07.124086 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:07.124046    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-f7wjz"
May 26 11:40:07.258587 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:07.258554    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-f7wjz"]
May 26 11:40:07.261066 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:40:07.261036    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e24b7e6_2b5e_4bf3_b8ae_e4605d9c5a75.slice/crio-2aac6fadb057a31aad4b5e9db05ddf2e399290de65184a4eed21c2ecdf5a0f6a WatchSource:0}: Error finding container 2aac6fadb057a31aad4b5e9db05ddf2e399290de65184a4eed21c2ecdf5a0f6a: Status 404 returned error can't find the container with id 2aac6fadb057a31aad4b5e9db05ddf2e399290de65184a4eed21c2ecdf5a0f6a
May 26 11:40:07.333901 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:07.333865    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/be4489bb-28af-4904-ac19-f491dde48310-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pxkg7\" (UID: \"be4489bb-28af-4904-ac19-f491dde48310\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"
May 26 11:40:07.336407 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:07.336384    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/be4489bb-28af-4904-ac19-f491dde48310-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pxkg7\" (UID: \"be4489bb-28af-4904-ac19-f491dde48310\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"
May 26 11:40:07.520381 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:07.520343    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"
May 26 11:40:07.606452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:07.606414    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-f7wjz" event={"ID":"8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75","Type":"ContainerStarted","Data":"2aac6fadb057a31aad4b5e9db05ddf2e399290de65184a4eed21c2ecdf5a0f6a"}
May 26 11:40:07.654428 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:07.654401    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"]
May 26 11:40:07.656589 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:40:07.656554    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4489bb_28af_4904_ac19_f491dde48310.slice/crio-df25009ee873f3ce892f6f9ab63bd836b2ba41ae0c1d1e5f20e304bf5a4c55df WatchSource:0}: Error finding container df25009ee873f3ce892f6f9ab63bd836b2ba41ae0c1d1e5f20e304bf5a4c55df: Status 404 returned error can't find the container with id df25009ee873f3ce892f6f9ab63bd836b2ba41ae0c1d1e5f20e304bf5a4c55df
May 26 11:40:08.610721 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:08.610677    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7" event={"ID":"be4489bb-28af-4904-ac19-f491dde48310","Type":"ContainerStarted","Data":"df25009ee873f3ce892f6f9ab63bd836b2ba41ae0c1d1e5f20e304bf5a4c55df"}
May 26 11:40:09.615118 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:09.615074    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-6fhjs" event={"ID":"20be1137-924b-435d-a28f-4877cc68a105","Type":"ContainerStarted","Data":"a92a14288acc4eb7773353842e19e11a3b420a08bca245f184ec8c9c8265689e"}
May 26 11:40:09.615568 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:09.615193    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-6fhjs"
May 26 11:40:09.635936 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:09.635885    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-6fhjs" podStartSLOduration=1.677943323 podStartE2EDuration="4.635871685s" podCreationTimestamp="2026-05-26 11:40:05 +0000 UTC" firstStartedPulling="2026-05-26 11:40:06.459702335 +0000 UTC m=+939.265540784" lastFinishedPulling="2026-05-26 11:40:09.417630704 +0000 UTC m=+942.223469146" observedRunningTime="2026-05-26 11:40:09.634504096 +0000 UTC m=+942.440342562" watchObservedRunningTime="2026-05-26 11:40:09.635871685 +0000 UTC m=+942.441710148"
May 26 11:40:11.624345 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:11.624299    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7" event={"ID":"be4489bb-28af-4904-ac19-f491dde48310","Type":"ContainerStarted","Data":"9cc726d1dee6f2742808e79e81608c35bf4f7655f34f9fa65998356702ef42a5"}
May 26 11:40:11.624719 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:11.624367    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"
May 26 11:40:11.641316 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:11.641251    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7" podStartSLOduration=3.52510654 podStartE2EDuration="6.641236987s" podCreationTimestamp="2026-05-26 11:40:05 +0000 UTC" firstStartedPulling="2026-05-26 11:40:07.657908235 +0000 UTC m=+940.463746682" lastFinishedPulling="2026-05-26 11:40:10.774038674 +0000 UTC m=+943.579877129" observedRunningTime="2026-05-26 11:40:11.640048413 +0000 UTC m=+944.445886859" watchObservedRunningTime="2026-05-26 11:40:11.641236987 +0000 UTC m=+944.447075450"
May 26 11:40:13.632661 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:13.632625    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-f7wjz" event={"ID":"8e24b7e6-2b5e-4bf3-b8ae-e4605d9c5a75","Type":"ContainerStarted","Data":"2b982f9bdd2d8210c6e1d1ad496809ad25f895702feb5aa5a1d7d7b0608e05e1"}
May 26 11:40:13.633108 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:13.632751    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-f7wjz"
May 26 11:40:13.650344 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:13.650292    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-f7wjz" podStartSLOduration=2.505369699 podStartE2EDuration="8.650264106s" podCreationTimestamp="2026-05-26 11:40:05 +0000 UTC" firstStartedPulling="2026-05-26 11:40:07.262405889 +0000 UTC m=+940.068244332" lastFinishedPulling="2026-05-26 11:40:13.407300286 +0000 UTC m=+946.213138739" observedRunningTime="2026-05-26 11:40:13.648452853 +0000 UTC m=+946.454291341" watchObservedRunningTime="2026-05-26 11:40:13.650264106 +0000 UTC m=+946.456102591"
May 26 11:40:22.632442 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:22.632414    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pxkg7"
May 26 11:40:26.603625 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:26.603594    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fzddk"
May 26 11:40:30.621150 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:30.621113    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-6fhjs"
May 26 11:40:34.638835 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:40:34.638806    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-f7wjz"
May 26 11:41:15.436859 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.436822    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-c7jht"]
May 26 11:41:15.440182 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.440160    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht"
May 26 11:41:15.442791 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.442764    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 26 11:41:15.442900 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.442826    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 26 11:41:15.442986 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.442963    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-mtbhq\""
May 26 11:41:15.443734 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.443714    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 26 11:41:15.451138 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.451112    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-c7jht"]
May 26 11:41:15.454869 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.454845    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-ckw25"]
May 26 11:41:15.458266 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.458244    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-ckw25"
May 26 11:41:15.460985 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.460965    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
May 26 11:41:15.461746 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.461731    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-8fn7g\""
May 26 11:41:15.471400 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.471379    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-ckw25"]
May 26 11:41:15.484196 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.484168    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-9dgds"]
May 26 11:41:15.488963 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.488945    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-9dgds"
May 26 11:41:15.492241 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.492212    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 26 11:41:15.492481 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.492466    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-w8jd4\""
May 26 11:41:15.498805 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.498784    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-9dgds"]
May 26 11:41:15.519609 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.519585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/859849a2-195b-4b91-9326-1b2393fcef0c-cert\") pod \"kserve-controller-manager-7cd9b78f5d-c7jht\" (UID: \"859849a2-195b-4b91-9326-1b2393fcef0c\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht"
May 26 11:41:15.519708 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.519627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncgkg\" (UniqueName: \"kubernetes.io/projected/859849a2-195b-4b91-9326-1b2393fcef0c-kube-api-access-ncgkg\") pod \"kserve-controller-manager-7cd9b78f5d-c7jht\" (UID: \"859849a2-195b-4b91-9326-1b2393fcef0c\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht"
May 26 11:41:15.519708 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.519662    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-244zx\" (UniqueName: \"kubernetes.io/projected/217ba037-a3c8-4c99-b59a-90a25b8b89aa-kube-api-access-244zx\") pod \"seaweedfs-79f457656b-9dgds\" (UID: \"217ba037-a3c8-4c99-b59a-90a25b8b89aa\") " pod="kserve/seaweedfs-79f457656b-9dgds"
May 26 11:41:15.519708 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.519696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct2bx\" (UniqueName: \"kubernetes.io/projected/79a85375-a60f-4f2e-b091-9f5b26139bd6-kube-api-access-ct2bx\") pod \"llmisvc-controller-manager-6b94ff949c-ckw25\" (UID: \"79a85375-a60f-4f2e-b091-9f5b26139bd6\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-ckw25"
May 26 11:41:15.519812 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.519746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79a85375-a60f-4f2e-b091-9f5b26139bd6-cert\") pod \"llmisvc-controller-manager-6b94ff949c-ckw25\" (UID: \"79a85375-a60f-4f2e-b091-9f5b26139bd6\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-ckw25"
May 26 11:41:15.519812 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.519804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/217ba037-a3c8-4c99-b59a-90a25b8b89aa-data\") pod \"seaweedfs-79f457656b-9dgds\" (UID: \"217ba037-a3c8-4c99-b59a-90a25b8b89aa\") " pod="kserve/seaweedfs-79f457656b-9dgds"
May 26 11:41:15.621156 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.621122    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/217ba037-a3c8-4c99-b59a-90a25b8b89aa-data\") pod \"seaweedfs-79f457656b-9dgds\" (UID: \"217ba037-a3c8-4c99-b59a-90a25b8b89aa\") " pod="kserve/seaweedfs-79f457656b-9dgds"
May 26 11:41:15.621365 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.621164    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/859849a2-195b-4b91-9326-1b2393fcef0c-cert\") pod \"kserve-controller-manager-7cd9b78f5d-c7jht\" (UID: \"859849a2-195b-4b91-9326-1b2393fcef0c\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht"
May 26 11:41:15.621365 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.621184    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ncgkg\" (UniqueName: \"kubernetes.io/projected/859849a2-195b-4b91-9326-1b2393fcef0c-kube-api-access-ncgkg\") pod \"kserve-controller-manager-7cd9b78f5d-c7jht\" (UID: \"859849a2-195b-4b91-9326-1b2393fcef0c\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht"
May 26 11:41:15.621365 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.621205    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-244zx\" (UniqueName: \"kubernetes.io/projected/217ba037-a3c8-4c99-b59a-90a25b8b89aa-kube-api-access-244zx\") pod \"seaweedfs-79f457656b-9dgds\" (UID: \"217ba037-a3c8-4c99-b59a-90a25b8b89aa\") " pod="kserve/seaweedfs-79f457656b-9dgds"
May 26 11:41:15.621365 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.621224    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ct2bx\" (UniqueName: \"kubernetes.io/projected/79a85375-a60f-4f2e-b091-9f5b26139bd6-kube-api-access-ct2bx\") pod \"llmisvc-controller-manager-6b94ff949c-ckw25\" (UID: \"79a85375-a60f-4f2e-b091-9f5b26139bd6\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-ckw25"
May 26 11:41:15.621365 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.621267    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79a85375-a60f-4f2e-b091-9f5b26139bd6-cert\") pod \"llmisvc-controller-manager-6b94ff949c-ckw25\" (UID: \"79a85375-a60f-4f2e-b091-9f5b26139bd6\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-ckw25"
May 26 11:41:15.621613 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.621528    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/217ba037-a3c8-4c99-b59a-90a25b8b89aa-data\") pod \"seaweedfs-79f457656b-9dgds\" (UID: \"217ba037-a3c8-4c99-b59a-90a25b8b89aa\") " pod="kserve/seaweedfs-79f457656b-9dgds"
May 26 11:41:15.623656 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.623633    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79a85375-a60f-4f2e-b091-9f5b26139bd6-cert\") pod \"llmisvc-controller-manager-6b94ff949c-ckw25\" (UID: \"79a85375-a60f-4f2e-b091-9f5b26139bd6\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-ckw25"
May 26 11:41:15.623759 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.623638    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/859849a2-195b-4b91-9326-1b2393fcef0c-cert\") pod \"kserve-controller-manager-7cd9b78f5d-c7jht\" (UID: \"859849a2-195b-4b91-9326-1b2393fcef0c\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht"
May 26 11:41:15.631628 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.631604    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct2bx\" (UniqueName: \"kubernetes.io/projected/79a85375-a60f-4f2e-b091-9f5b26139bd6-kube-api-access-ct2bx\") pod \"llmisvc-controller-manager-6b94ff949c-ckw25\" (UID: \"79a85375-a60f-4f2e-b091-9f5b26139bd6\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-ckw25"
May 26 11:41:15.631732 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.631641    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-244zx\" (UniqueName: \"kubernetes.io/projected/217ba037-a3c8-4c99-b59a-90a25b8b89aa-kube-api-access-244zx\") pod \"seaweedfs-79f457656b-9dgds\" (UID: \"217ba037-a3c8-4c99-b59a-90a25b8b89aa\") " pod="kserve/seaweedfs-79f457656b-9dgds"
May 26 11:41:15.631781 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.631736    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncgkg\" (UniqueName: \"kubernetes.io/projected/859849a2-195b-4b91-9326-1b2393fcef0c-kube-api-access-ncgkg\") pod \"kserve-controller-manager-7cd9b78f5d-c7jht\" (UID: \"859849a2-195b-4b91-9326-1b2393fcef0c\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht"
May 26 11:41:15.752125 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.752093    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht"
May 26 11:41:15.770995 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.770955    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-ckw25"
May 26 11:41:15.800263 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.800143    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-9dgds"
May 26 11:41:15.905103 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.905053    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-c7jht"]
May 26 11:41:15.908581 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:41:15.908545    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859849a2_195b_4b91_9326_1b2393fcef0c.slice/crio-3ec87765736ba9795bdbfbd8885829fbf942237e8911186d3b98adbcae042584 WatchSource:0}: Error finding container 3ec87765736ba9795bdbfbd8885829fbf942237e8911186d3b98adbcae042584: Status 404 returned error can't find the container with id 3ec87765736ba9795bdbfbd8885829fbf942237e8911186d3b98adbcae042584
May 26 11:41:15.933370 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.930588    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-ckw25"]
May 26 11:41:15.959542 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:15.959518    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-9dgds"]
May 26 11:41:15.961896 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:41:15.961864    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217ba037_a3c8_4c99_b59a_90a25b8b89aa.slice/crio-1b7149c1dc41fc006a87143984852690efd4780757abe56969bc4ee70478ac61 WatchSource:0}: Error finding container 1b7149c1dc41fc006a87143984852690efd4780757abe56969bc4ee70478ac61: Status 404 returned error can't find the container with id 1b7149c1dc41fc006a87143984852690efd4780757abe56969bc4ee70478ac61
May 26 11:41:16.838843 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:16.838798    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-ckw25" event={"ID":"79a85375-a60f-4f2e-b091-9f5b26139bd6","Type":"ContainerStarted","Data":"cdf9dc53257514b93ea4e3e75926bb70ba19e9eca856d63857bc48eac64e3c08"}
May 26 11:41:16.840671 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:16.840601    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht" event={"ID":"859849a2-195b-4b91-9326-1b2393fcef0c","Type":"ContainerStarted","Data":"3ec87765736ba9795bdbfbd8885829fbf942237e8911186d3b98adbcae042584"}
May 26 11:41:16.842552 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:16.842510    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-9dgds" event={"ID":"217ba037-a3c8-4c99-b59a-90a25b8b89aa","Type":"ContainerStarted","Data":"1b7149c1dc41fc006a87143984852690efd4780757abe56969bc4ee70478ac61"}
May 26 11:41:20.859695 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:20.859586    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-ckw25" event={"ID":"79a85375-a60f-4f2e-b091-9f5b26139bd6","Type":"ContainerStarted","Data":"eb4920db8f1ed598ecaa667599a86b9c1c3b33cf70f42f7530040e6eccb4cc65"}
May 26 11:41:20.859695 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:20.859639    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-ckw25"
May 26 11:41:20.860788 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:20.860762    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht" event={"ID":"859849a2-195b-4b91-9326-1b2393fcef0c","Type":"ContainerStarted","Data":"1b3566888d16e183e4b50e09dad43c4c7f5c8f222f1456733e3b1467e9005fce"}
May 26 11:41:20.860962 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:20.860886    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht"
May 26 11:41:20.862006 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:20.861982    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-9dgds" event={"ID":"217ba037-a3c8-4c99-b59a-90a25b8b89aa","Type":"ContainerStarted","Data":"9bbc88c6c25c6ae678b5875372b062e4905312c9b9bc241c9a49e9d3255dc029"}
May 26 11:41:20.862103 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:20.862084    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-9dgds"
May 26 11:41:20.875915 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:20.875874    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6b94ff949c-ckw25" podStartSLOduration=1.318961836 podStartE2EDuration="5.875860787s" podCreationTimestamp="2026-05-26 11:41:15 +0000 UTC" firstStartedPulling="2026-05-26 11:41:15.934313329 +0000 UTC m=+1008.740151773" lastFinishedPulling="2026-05-26 11:41:20.491212282 +0000 UTC m=+1013.297050724" observedRunningTime="2026-05-26 11:41:20.875122917 +0000 UTC m=+1013.680961383" watchObservedRunningTime="2026-05-26 11:41:20.875860787 +0000 UTC m=+1013.681699250"
May 26 11:41:20.895378 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:20.895329    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-9dgds" podStartSLOduration=1.315985842 podStartE2EDuration="5.895314647s" podCreationTimestamp="2026-05-26 11:41:15 +0000 UTC" firstStartedPulling="2026-05-26 11:41:15.96304536 +0000 UTC m=+1008.768883802" lastFinishedPulling="2026-05-26 11:41:20.542374155 +0000 UTC m=+1013.348212607" observedRunningTime="2026-05-26 11:41:20.894258839 +0000 UTC m=+1013.700097303" watchObservedRunningTime="2026-05-26 11:41:20.895314647 +0000 UTC m=+1013.701153110"
May 26 11:41:20.910701 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:20.910645    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht" podStartSLOduration=1.509858785 podStartE2EDuration="5.910630255s" podCreationTimestamp="2026-05-26 11:41:15 +0000 UTC" firstStartedPulling="2026-05-26 11:41:15.91023092 +0000 UTC m=+1008.716069362" lastFinishedPulling="2026-05-26 11:41:20.311002379 +0000 UTC m=+1013.116840832" observedRunningTime="2026-05-26 11:41:20.910185792 +0000 UTC m=+1013.716024267" watchObservedRunningTime="2026-05-26 11:41:20.910630255 +0000 UTC m=+1013.716468717"
May 26 11:41:26.867673 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:26.867640    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-9dgds"
May 26 11:41:51.867431 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:51.867395    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-ckw25"
May 26 11:41:51.870402 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:51.870381    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht"
May 26 11:41:53.234544 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.234509    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-c7jht"]
May 26 11:41:53.234935 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.234720    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht" podUID="859849a2-195b-4b91-9326-1b2393fcef0c" containerName="manager" containerID="cri-o://1b3566888d16e183e4b50e09dad43c4c7f5c8f222f1456733e3b1467e9005fce" gracePeriod=10
May 26 11:41:53.253033 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.253009    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-skfch"]
May 26 11:41:53.255299 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.255261    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-skfch"
May 26 11:41:53.265028 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.265007    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-skfch"]
May 26 11:41:53.353161 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.353134    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgxxx\" (UniqueName: \"kubernetes.io/projected/55c5bd52-67a2-4519-a10d-caa351829609-kube-api-access-jgxxx\") pod \"kserve-controller-manager-7cd9b78f5d-skfch\" (UID: \"55c5bd52-67a2-4519-a10d-caa351829609\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-skfch"
May 26 11:41:53.353254 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.353189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55c5bd52-67a2-4519-a10d-caa351829609-cert\") pod \"kserve-controller-manager-7cd9b78f5d-skfch\" (UID: \"55c5bd52-67a2-4519-a10d-caa351829609\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-skfch"
May 26 11:41:53.454074 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.454041    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jgxxx\" (UniqueName: \"kubernetes.io/projected/55c5bd52-67a2-4519-a10d-caa351829609-kube-api-access-jgxxx\") pod \"kserve-controller-manager-7cd9b78f5d-skfch\" (UID: \"55c5bd52-67a2-4519-a10d-caa351829609\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-skfch"
May 26 11:41:53.454168 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.454123    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55c5bd52-67a2-4519-a10d-caa351829609-cert\") pod \"kserve-controller-manager-7cd9b78f5d-skfch\" (UID: \"55c5bd52-67a2-4519-a10d-caa351829609\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-skfch"
May 26 11:41:53.456398 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.456373    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55c5bd52-67a2-4519-a10d-caa351829609-cert\") pod \"kserve-controller-manager-7cd9b78f5d-skfch\" (UID: \"55c5bd52-67a2-4519-a10d-caa351829609\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-skfch"
May 26 11:41:53.462986 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.462961    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgxxx\" (UniqueName: \"kubernetes.io/projected/55c5bd52-67a2-4519-a10d-caa351829609-kube-api-access-jgxxx\") pod \"kserve-controller-manager-7cd9b78f5d-skfch\" (UID: \"55c5bd52-67a2-4519-a10d-caa351829609\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-skfch"
May 26 11:41:53.477191 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.477166    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht"
May 26 11:41:53.555073 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.554987    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/859849a2-195b-4b91-9326-1b2393fcef0c-cert\") pod \"859849a2-195b-4b91-9326-1b2393fcef0c\" (UID: \"859849a2-195b-4b91-9326-1b2393fcef0c\") "
May 26 11:41:53.555073 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.555035    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncgkg\" (UniqueName: \"kubernetes.io/projected/859849a2-195b-4b91-9326-1b2393fcef0c-kube-api-access-ncgkg\") pod \"859849a2-195b-4b91-9326-1b2393fcef0c\" (UID: \"859849a2-195b-4b91-9326-1b2393fcef0c\") "
May 26 11:41:53.557243 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.557214    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/859849a2-195b-4b91-9326-1b2393fcef0c-cert" (OuterVolumeSpecName: "cert") pod "859849a2-195b-4b91-9326-1b2393fcef0c" (UID: "859849a2-195b-4b91-9326-1b2393fcef0c"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:41:53.557390 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.557245    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859849a2-195b-4b91-9326-1b2393fcef0c-kube-api-access-ncgkg" (OuterVolumeSpecName: "kube-api-access-ncgkg") pod "859849a2-195b-4b91-9326-1b2393fcef0c" (UID: "859849a2-195b-4b91-9326-1b2393fcef0c"). InnerVolumeSpecName "kube-api-access-ncgkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:41:53.599633 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.599577    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-skfch"
May 26 11:41:53.656109 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.656049    2563 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/859849a2-195b-4b91-9326-1b2393fcef0c-cert\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:41:53.656109 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.656083    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncgkg\" (UniqueName: \"kubernetes.io/projected/859849a2-195b-4b91-9326-1b2393fcef0c-kube-api-access-ncgkg\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:41:53.728292 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.728254    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-skfch"]
May 26 11:41:53.730905 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:41:53.730874    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c5bd52_67a2_4519_a10d_caa351829609.slice/crio-d6764438d5394b399a6837a0014e207e3e285dfbf0227685d0ef6363941f5b12 WatchSource:0}: Error finding container d6764438d5394b399a6837a0014e207e3e285dfbf0227685d0ef6363941f5b12: Status 404 returned error can't find the container with id d6764438d5394b399a6837a0014e207e3e285dfbf0227685d0ef6363941f5b12
May 26 11:41:53.966532 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.966448    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-skfch" event={"ID":"55c5bd52-67a2-4519-a10d-caa351829609","Type":"ContainerStarted","Data":"d6764438d5394b399a6837a0014e207e3e285dfbf0227685d0ef6363941f5b12"}
May 26 11:41:53.967380 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.967361    2563 generic.go:358] "Generic (PLEG): container finished" podID="859849a2-195b-4b91-9326-1b2393fcef0c" containerID="1b3566888d16e183e4b50e09dad43c4c7f5c8f222f1456733e3b1467e9005fce" exitCode=0
May 26 11:41:53.967427 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.967412    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht" event={"ID":"859849a2-195b-4b91-9326-1b2393fcef0c","Type":"ContainerDied","Data":"1b3566888d16e183e4b50e09dad43c4c7f5c8f222f1456733e3b1467e9005fce"}
May 26 11:41:53.967467 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.967433    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht" event={"ID":"859849a2-195b-4b91-9326-1b2393fcef0c","Type":"ContainerDied","Data":"3ec87765736ba9795bdbfbd8885829fbf942237e8911186d3b98adbcae042584"}
May 26 11:41:53.967467 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.967447    2563 scope.go:117] "RemoveContainer" containerID="1b3566888d16e183e4b50e09dad43c4c7f5c8f222f1456733e3b1467e9005fce"
May 26 11:41:53.967467 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.967447    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-c7jht"
May 26 11:41:53.974863 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.974838    2563 scope.go:117] "RemoveContainer" containerID="1b3566888d16e183e4b50e09dad43c4c7f5c8f222f1456733e3b1467e9005fce"
May 26 11:41:53.975116 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:41:53.975090    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3566888d16e183e4b50e09dad43c4c7f5c8f222f1456733e3b1467e9005fce\": container with ID starting with 1b3566888d16e183e4b50e09dad43c4c7f5c8f222f1456733e3b1467e9005fce not found: ID does not exist" containerID="1b3566888d16e183e4b50e09dad43c4c7f5c8f222f1456733e3b1467e9005fce"
May 26 11:41:53.975171 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.975120    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3566888d16e183e4b50e09dad43c4c7f5c8f222f1456733e3b1467e9005fce"} err="failed to get container status \"1b3566888d16e183e4b50e09dad43c4c7f5c8f222f1456733e3b1467e9005fce\": rpc error: code = NotFound desc = could not find container \"1b3566888d16e183e4b50e09dad43c4c7f5c8f222f1456733e3b1467e9005fce\": container with ID starting with 1b3566888d16e183e4b50e09dad43c4c7f5c8f222f1456733e3b1467e9005fce not found: ID does not exist"
May 26 11:41:53.986093 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.986071    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-c7jht"]
May 26 11:41:53.992112 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:53.992089    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-c7jht"]
May 26 11:41:54.971912 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:54.971877    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-skfch" event={"ID":"55c5bd52-67a2-4519-a10d-caa351829609","Type":"ContainerStarted","Data":"972da5cffc21cd8dff0a24eee0c37bb43baca60bec4777d2330a8c6cd600b92c"}
May 26 11:41:54.972319 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:54.971931    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7cd9b78f5d-skfch"
May 26 11:41:54.988606 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:54.988481    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7cd9b78f5d-skfch" podStartSLOduration=1.644258082 podStartE2EDuration="1.988465369s" podCreationTimestamp="2026-05-26 11:41:53 +0000 UTC" firstStartedPulling="2026-05-26 11:41:53.732219568 +0000 UTC m=+1046.538058011" lastFinishedPulling="2026-05-26 11:41:54.076426853 +0000 UTC m=+1046.882265298" observedRunningTime="2026-05-26 11:41:54.987937827 +0000 UTC m=+1047.793776295" watchObservedRunningTime="2026-05-26 11:41:54.988465369 +0000 UTC m=+1047.794303834"
May 26 11:41:55.796220 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:41:55.796188    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="859849a2-195b-4b91-9326-1b2393fcef0c" path="/var/lib/kubelet/pods/859849a2-195b-4b91-9326-1b2393fcef0c/volumes"
May 26 11:42:25.981675 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:25.981638    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7cd9b78f5d-skfch"
May 26 11:42:26.896518 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:26.896480    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-q622l"]
May 26 11:42:26.896851 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:26.896839    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="859849a2-195b-4b91-9326-1b2393fcef0c" containerName="manager"
May 26 11:42:26.896896 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:26.896852    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="859849a2-195b-4b91-9326-1b2393fcef0c" containerName="manager"
May 26 11:42:26.896933 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:26.896916    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="859849a2-195b-4b91-9326-1b2393fcef0c" containerName="manager"
May 26 11:42:26.899391 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:26.899375    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-q622l"
May 26 11:42:26.902495 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:26.902473    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-wpvwj\""
May 26 11:42:26.902945 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:26.902923    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 26 11:42:26.922797 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:26.922766    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-q622l"]
May 26 11:42:26.941493 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:26.941466    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-bwrpv"]
May 26 11:42:26.943804 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:26.943788    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-bwrpv"
May 26 11:42:26.946386 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:26.946362    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 26 11:42:26.946510 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:26.946365    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-67xwv\""
May 26 11:42:26.959717 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:26.959690    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-bwrpv"]
May 26 11:42:27.048924 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.048884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca12c3cc-0728-498d-b182-d489f05cf877-cert\") pod \"odh-model-controller-696fc77849-bwrpv\" (UID: \"ca12c3cc-0728-498d-b182-d489f05cf877\") " pod="kserve/odh-model-controller-696fc77849-bwrpv"
May 26 11:42:27.049402 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.048934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ad2ceb-5698-46bd-aaee-6a8b0b5e96b8-tls-certs\") pod \"model-serving-api-86f7b4b499-q622l\" (UID: \"52ad2ceb-5698-46bd-aaee-6a8b0b5e96b8\") " pod="kserve/model-serving-api-86f7b4b499-q622l"
May 26 11:42:27.049402 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.049005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hrsq\" (UniqueName: \"kubernetes.io/projected/ca12c3cc-0728-498d-b182-d489f05cf877-kube-api-access-5hrsq\") pod \"odh-model-controller-696fc77849-bwrpv\" (UID: \"ca12c3cc-0728-498d-b182-d489f05cf877\") " pod="kserve/odh-model-controller-696fc77849-bwrpv"
May 26 11:42:27.049402 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.049055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkscb\" (UniqueName: \"kubernetes.io/projected/52ad2ceb-5698-46bd-aaee-6a8b0b5e96b8-kube-api-access-gkscb\") pod \"model-serving-api-86f7b4b499-q622l\" (UID: \"52ad2ceb-5698-46bd-aaee-6a8b0b5e96b8\") " pod="kserve/model-serving-api-86f7b4b499-q622l"
May 26 11:42:27.150390 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.150310    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca12c3cc-0728-498d-b182-d489f05cf877-cert\") pod \"odh-model-controller-696fc77849-bwrpv\" (UID: \"ca12c3cc-0728-498d-b182-d489f05cf877\") " pod="kserve/odh-model-controller-696fc77849-bwrpv"
May 26 11:42:27.150390 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.150360    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ad2ceb-5698-46bd-aaee-6a8b0b5e96b8-tls-certs\") pod \"model-serving-api-86f7b4b499-q622l\" (UID: \"52ad2ceb-5698-46bd-aaee-6a8b0b5e96b8\") " pod="kserve/model-serving-api-86f7b4b499-q622l"
May 26 11:42:27.150390 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.150398    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5hrsq\" (UniqueName: \"kubernetes.io/projected/ca12c3cc-0728-498d-b182-d489f05cf877-kube-api-access-5hrsq\") pod \"odh-model-controller-696fc77849-bwrpv\" (UID: \"ca12c3cc-0728-498d-b182-d489f05cf877\") " pod="kserve/odh-model-controller-696fc77849-bwrpv"
May 26 11:42:27.150657 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.150434    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gkscb\" (UniqueName: \"kubernetes.io/projected/52ad2ceb-5698-46bd-aaee-6a8b0b5e96b8-kube-api-access-gkscb\") pod \"model-serving-api-86f7b4b499-q622l\" (UID: \"52ad2ceb-5698-46bd-aaee-6a8b0b5e96b8\") " pod="kserve/model-serving-api-86f7b4b499-q622l"
May 26 11:42:27.150657 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:42:27.150465    2563 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
May 26 11:42:27.150657 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:42:27.150544    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca12c3cc-0728-498d-b182-d489f05cf877-cert podName:ca12c3cc-0728-498d-b182-d489f05cf877 nodeName:}" failed. No retries permitted until 2026-05-26 11:42:27.650521654 +0000 UTC m=+1080.456360099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ca12c3cc-0728-498d-b182-d489f05cf877-cert") pod "odh-model-controller-696fc77849-bwrpv" (UID: "ca12c3cc-0728-498d-b182-d489f05cf877") : secret "odh-model-controller-webhook-cert" not found
May 26 11:42:27.152876 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.152853    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ad2ceb-5698-46bd-aaee-6a8b0b5e96b8-tls-certs\") pod \"model-serving-api-86f7b4b499-q622l\" (UID: \"52ad2ceb-5698-46bd-aaee-6a8b0b5e96b8\") " pod="kserve/model-serving-api-86f7b4b499-q622l"
May 26 11:42:27.159590 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.159564    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hrsq\" (UniqueName: \"kubernetes.io/projected/ca12c3cc-0728-498d-b182-d489f05cf877-kube-api-access-5hrsq\") pod \"odh-model-controller-696fc77849-bwrpv\" (UID: \"ca12c3cc-0728-498d-b182-d489f05cf877\") " pod="kserve/odh-model-controller-696fc77849-bwrpv"
May 26 11:42:27.159699 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.159594    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkscb\" (UniqueName: \"kubernetes.io/projected/52ad2ceb-5698-46bd-aaee-6a8b0b5e96b8-kube-api-access-gkscb\") pod \"model-serving-api-86f7b4b499-q622l\" (UID: \"52ad2ceb-5698-46bd-aaee-6a8b0b5e96b8\") " pod="kserve/model-serving-api-86f7b4b499-q622l"
May 26 11:42:27.210188 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.210152    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-q622l"
May 26 11:42:27.340629 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.340599    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-q622l"]
May 26 11:42:27.343213 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:42:27.343179    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ad2ceb_5698_46bd_aaee_6a8b0b5e96b8.slice/crio-ee7dc564764edabd33d217a20bc73049bdb3b5a9e170c9a19a18bb19ccb882ff WatchSource:0}: Error finding container ee7dc564764edabd33d217a20bc73049bdb3b5a9e170c9a19a18bb19ccb882ff: Status 404 returned error can't find the container with id ee7dc564764edabd33d217a20bc73049bdb3b5a9e170c9a19a18bb19ccb882ff
May 26 11:42:27.654528 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.654494    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca12c3cc-0728-498d-b182-d489f05cf877-cert\") pod \"odh-model-controller-696fc77849-bwrpv\" (UID: \"ca12c3cc-0728-498d-b182-d489f05cf877\") " pod="kserve/odh-model-controller-696fc77849-bwrpv"
May 26 11:42:27.656928 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.656899    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca12c3cc-0728-498d-b182-d489f05cf877-cert\") pod \"odh-model-controller-696fc77849-bwrpv\" (UID: \"ca12c3cc-0728-498d-b182-d489f05cf877\") " pod="kserve/odh-model-controller-696fc77849-bwrpv"
May 26 11:42:27.858452 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.858421    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-67xwv\""
May 26 11:42:27.866599 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:27.866512    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-bwrpv"
May 26 11:42:28.029130 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:28.029086    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-bwrpv"]
May 26 11:42:28.030998 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:42:28.030944    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca12c3cc_0728_498d_b182_d489f05cf877.slice/crio-72e1240b36c99edfd9f7377f1e7dc7cd5c4a46b8cb7bab9b5dbcd09be6201667 WatchSource:0}: Error finding container 72e1240b36c99edfd9f7377f1e7dc7cd5c4a46b8cb7bab9b5dbcd09be6201667: Status 404 returned error can't find the container with id 72e1240b36c99edfd9f7377f1e7dc7cd5c4a46b8cb7bab9b5dbcd09be6201667
May 26 11:42:28.087220 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:28.087179    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-bwrpv" event={"ID":"ca12c3cc-0728-498d-b182-d489f05cf877","Type":"ContainerStarted","Data":"72e1240b36c99edfd9f7377f1e7dc7cd5c4a46b8cb7bab9b5dbcd09be6201667"}
May 26 11:42:28.089300 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:28.089252    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-q622l" event={"ID":"52ad2ceb-5698-46bd-aaee-6a8b0b5e96b8","Type":"ContainerStarted","Data":"ee7dc564764edabd33d217a20bc73049bdb3b5a9e170c9a19a18bb19ccb882ff"}
May 26 11:42:31.110375 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:31.110339    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-bwrpv" event={"ID":"ca12c3cc-0728-498d-b182-d489f05cf877","Type":"ContainerStarted","Data":"df9eaad15ba8ad011cbbc8dad0946f496c1533259fac3525169645f83d87c514"}
May 26 11:42:31.110813 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:31.110498    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-bwrpv"
May 26 11:42:31.111705 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:31.111681    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-q622l" event={"ID":"52ad2ceb-5698-46bd-aaee-6a8b0b5e96b8","Type":"ContainerStarted","Data":"78e4444c0e2b6758006fcf9d4cfbddb3e2b194b088a30759f562618eec06179b"}
May 26 11:42:31.111811 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:31.111801    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-q622l"
May 26 11:42:31.143862 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:31.143823    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-bwrpv" podStartSLOduration=2.209369588 podStartE2EDuration="5.143810223s" podCreationTimestamp="2026-05-26 11:42:26 +0000 UTC" firstStartedPulling="2026-05-26 11:42:28.03274454 +0000 UTC m=+1080.838582983" lastFinishedPulling="2026-05-26 11:42:30.967185176 +0000 UTC m=+1083.773023618" observedRunningTime="2026-05-26 11:42:31.141394047 +0000 UTC m=+1083.947232524" watchObservedRunningTime="2026-05-26 11:42:31.143810223 +0000 UTC m=+1083.949648687"
May 26 11:42:31.166692 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:31.166643    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-q622l" podStartSLOduration=1.547654547 podStartE2EDuration="5.166629292s" podCreationTimestamp="2026-05-26 11:42:26 +0000 UTC" firstStartedPulling="2026-05-26 11:42:27.344971184 +0000 UTC m=+1080.150809630" lastFinishedPulling="2026-05-26 11:42:30.963945932 +0000 UTC m=+1083.769784375" observedRunningTime="2026-05-26 11:42:31.165364853 +0000 UTC m=+1083.971203318" watchObservedRunningTime="2026-05-26 11:42:31.166629292 +0000 UTC m=+1083.972467756"
May 26 11:42:42.116840 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:42.116808    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-bwrpv"
May 26 11:42:42.118547 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:42:42.118517    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-q622l"
May 26 11:43:02.656303 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:02.656255    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"]
May 26 11:43:02.659238 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:02.659216    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:43:02.661640 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:02.661615    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-378f2-predictor-serving-cert\""
May 26 11:43:02.661765 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:02.661615    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-378f2-kube-rbac-proxy-sar-config\""
May 26 11:43:02.661912 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:02.661896    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7tgml\""
May 26 11:43:02.662580 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:02.662560    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 11:43:02.662670 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:02.662569    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 11:43:02.667271 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:02.667246    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"]
May 26 11:43:02.737004 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:02.736966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-proxy-tls\") pod \"success-200-isvc-378f2-predictor-8bbd48d54-vp5rd\" (UID: \"8c56111a-f26e-4266-bb10-86bdcdf8f2d7\") " pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:43:02.737197 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:02.737031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-378f2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-success-200-isvc-378f2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-378f2-predictor-8bbd48d54-vp5rd\" (UID: \"8c56111a-f26e-4266-bb10-86bdcdf8f2d7\") " pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:43:02.737197 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:02.737061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtbmd\" (UniqueName: \"kubernetes.io/projected/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-kube-api-access-dtbmd\") pod \"success-200-isvc-378f2-predictor-8bbd48d54-vp5rd\" (UID: \"8c56111a-f26e-4266-bb10-86bdcdf8f2d7\") " pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:43:02.838224 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:02.838187    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-proxy-tls\") pod \"success-200-isvc-378f2-predictor-8bbd48d54-vp5rd\" (UID: \"8c56111a-f26e-4266-bb10-86bdcdf8f2d7\") " pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:43:02.838435 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:02.838253    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-378f2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-success-200-isvc-378f2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-378f2-predictor-8bbd48d54-vp5rd\" (UID: \"8c56111a-f26e-4266-bb10-86bdcdf8f2d7\") " pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:43:02.838435 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:02.838314    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dtbmd\" (UniqueName: \"kubernetes.io/projected/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-kube-api-access-dtbmd\") pod \"success-200-isvc-378f2-predictor-8bbd48d54-vp5rd\" (UID: \"8c56111a-f26e-4266-bb10-86bdcdf8f2d7\") " pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:43:02.838435 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:43:02.838415    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-378f2-predictor-serving-cert: secret "success-200-isvc-378f2-predictor-serving-cert" not found
May 26 11:43:02.838608 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:43:02.838502    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-proxy-tls podName:8c56111a-f26e-4266-bb10-86bdcdf8f2d7 nodeName:}" failed. No retries permitted until 2026-05-26 11:43:03.338479511 +0000 UTC m=+1116.144317955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-proxy-tls") pod "success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" (UID: "8c56111a-f26e-4266-bb10-86bdcdf8f2d7") : secret "success-200-isvc-378f2-predictor-serving-cert" not found
May 26 11:43:02.839339 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:02.839313    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-378f2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-success-200-isvc-378f2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-378f2-predictor-8bbd48d54-vp5rd\" (UID: \"8c56111a-f26e-4266-bb10-86bdcdf8f2d7\") " pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:43:02.849465 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:02.849437    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtbmd\" (UniqueName: \"kubernetes.io/projected/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-kube-api-access-dtbmd\") pod \"success-200-isvc-378f2-predictor-8bbd48d54-vp5rd\" (UID: \"8c56111a-f26e-4266-bb10-86bdcdf8f2d7\") " pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:43:03.342159 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:03.342117    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-proxy-tls\") pod \"success-200-isvc-378f2-predictor-8bbd48d54-vp5rd\" (UID: \"8c56111a-f26e-4266-bb10-86bdcdf8f2d7\") " pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:43:03.344667 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:03.344638    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-proxy-tls\") pod \"success-200-isvc-378f2-predictor-8bbd48d54-vp5rd\" (UID: \"8c56111a-f26e-4266-bb10-86bdcdf8f2d7\") " pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:43:03.572828 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:03.572784    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:43:03.709273 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:03.709242    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"]
May 26 11:43:03.711872 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:43:03.711842    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c56111a_f26e_4266_bb10_86bdcdf8f2d7.slice/crio-67154915a93b5b5b8d545d15a907ae50cab266d4c009edf357739e8a596cd336 WatchSource:0}: Error finding container 67154915a93b5b5b8d545d15a907ae50cab266d4c009edf357739e8a596cd336: Status 404 returned error can't find the container with id 67154915a93b5b5b8d545d15a907ae50cab266d4c009edf357739e8a596cd336
May 26 11:43:03.825893 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:03.825854    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"]
May 26 11:43:03.829714 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:03.829695    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:03.832034 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:03.832012    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
May 26 11:43:03.832166 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:03.832066    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
May 26 11:43:03.838343 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:03.838298    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"]
May 26 11:43:03.947998 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:03.947898    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9dff39d-f2cb-48e9-a3c1-3753107c084c-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:03.947998 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:03.947953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9dff39d-f2cb-48e9-a3c1-3753107c084c-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:03.947998 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:03.947982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hkqb\" (UniqueName: \"kubernetes.io/projected/e9dff39d-f2cb-48e9-a3c1-3753107c084c-kube-api-access-8hkqb\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:03.948247 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:03.948111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9dff39d-f2cb-48e9-a3c1-3753107c084c-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:04.049269 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:04.049234    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9dff39d-f2cb-48e9-a3c1-3753107c084c-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:04.049468 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:04.049298    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9dff39d-f2cb-48e9-a3c1-3753107c084c-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:04.049468 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:04.049329    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8hkqb\" (UniqueName: \"kubernetes.io/projected/e9dff39d-f2cb-48e9-a3c1-3753107c084c-kube-api-access-8hkqb\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:04.049468 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:04.049373    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9dff39d-f2cb-48e9-a3c1-3753107c084c-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:04.049626 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:43:04.049526    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
May 26 11:43:04.049626 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:43:04.049598    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9dff39d-f2cb-48e9-a3c1-3753107c084c-proxy-tls podName:e9dff39d-f2cb-48e9-a3c1-3753107c084c nodeName:}" failed. No retries permitted until 2026-05-26 11:43:04.549575933 +0000 UTC m=+1117.355414387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e9dff39d-f2cb-48e9-a3c1-3753107c084c-proxy-tls") pod "isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" (UID: "e9dff39d-f2cb-48e9-a3c1-3753107c084c") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
May 26 11:43:04.049773 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:04.049752    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9dff39d-f2cb-48e9-a3c1-3753107c084c-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:04.050027 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:04.050007    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9dff39d-f2cb-48e9-a3c1-3753107c084c-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:04.057876 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:04.057853    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hkqb\" (UniqueName: \"kubernetes.io/projected/e9dff39d-f2cb-48e9-a3c1-3753107c084c-kube-api-access-8hkqb\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:04.223150 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:04.223055    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" event={"ID":"8c56111a-f26e-4266-bb10-86bdcdf8f2d7","Type":"ContainerStarted","Data":"67154915a93b5b5b8d545d15a907ae50cab266d4c009edf357739e8a596cd336"}
May 26 11:43:04.555808 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:04.555114    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9dff39d-f2cb-48e9-a3c1-3753107c084c-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:04.555808 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:43:04.555369    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
May 26 11:43:04.555808 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:43:04.555443    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9dff39d-f2cb-48e9-a3c1-3753107c084c-proxy-tls podName:e9dff39d-f2cb-48e9-a3c1-3753107c084c nodeName:}" failed. No retries permitted until 2026-05-26 11:43:05.55542213 +0000 UTC m=+1118.361260576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e9dff39d-f2cb-48e9-a3c1-3753107c084c-proxy-tls") pod "isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" (UID: "e9dff39d-f2cb-48e9-a3c1-3753107c084c") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
May 26 11:43:05.567274 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:05.567235    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9dff39d-f2cb-48e9-a3c1-3753107c084c-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:05.571580 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:05.571522    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9dff39d-f2cb-48e9-a3c1-3753107c084c-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:05.643414 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:05.643241    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:05.854458 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:05.854371    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"]
May 26 11:43:05.861678 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:43:05.861645    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9dff39d_f2cb_48e9_a3c1_3753107c084c.slice/crio-da7172eaae3d145fa883b03691b0a1b4e7d2d492ad0f44bfe458146985bb4a72 WatchSource:0}: Error finding container da7172eaae3d145fa883b03691b0a1b4e7d2d492ad0f44bfe458146985bb4a72: Status 404 returned error can't find the container with id da7172eaae3d145fa883b03691b0a1b4e7d2d492ad0f44bfe458146985bb4a72
May 26 11:43:06.233872 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:06.233832    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" event={"ID":"e9dff39d-f2cb-48e9-a3c1-3753107c084c","Type":"ContainerStarted","Data":"da7172eaae3d145fa883b03691b0a1b4e7d2d492ad0f44bfe458146985bb4a72"}
May 26 11:43:16.274023 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:16.273971    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" event={"ID":"8c56111a-f26e-4266-bb10-86bdcdf8f2d7","Type":"ContainerStarted","Data":"591fbd77eda6419fe38d8a1577b0d1af109e312b01a04438d133559765a4731a"}
May 26 11:43:16.275704 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:16.275675    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" event={"ID":"e9dff39d-f2cb-48e9-a3c1-3753107c084c","Type":"ContainerStarted","Data":"8bdfadfce1d73c9eab5ac1e8547786a4153463d709dd1e4b95639e62d165f5d5"}
May 26 11:43:18.284342 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:18.284272    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" event={"ID":"8c56111a-f26e-4266-bb10-86bdcdf8f2d7","Type":"ContainerStarted","Data":"e24329d53edb13db0120e0f393022bb7a3b5798467edb9ca65b0be9857c26530"}
May 26 11:43:18.284802 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:18.284429    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:43:18.284802 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:18.284537    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:43:18.285745 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:18.285720    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 26 11:43:18.303327 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:18.303252    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" podStartSLOduration=2.189209036 podStartE2EDuration="16.303238769s" podCreationTimestamp="2026-05-26 11:43:02 +0000 UTC" firstStartedPulling="2026-05-26 11:43:03.713733078 +0000 UTC m=+1116.519571521" lastFinishedPulling="2026-05-26 11:43:17.827762812 +0000 UTC m=+1130.633601254" observedRunningTime="2026-05-26 11:43:18.302825114 +0000 UTC m=+1131.108663579" watchObservedRunningTime="2026-05-26 11:43:18.303238769 +0000 UTC m=+1131.109077234"
May 26 11:43:19.289358 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:19.289325    2563 generic.go:358] "Generic (PLEG): container finished" podID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerID="8bdfadfce1d73c9eab5ac1e8547786a4153463d709dd1e4b95639e62d165f5d5" exitCode=0
May 26 11:43:19.289729 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:19.289373    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" event={"ID":"e9dff39d-f2cb-48e9-a3c1-3753107c084c","Type":"ContainerDied","Data":"8bdfadfce1d73c9eab5ac1e8547786a4153463d709dd1e4b95639e62d165f5d5"}
May 26 11:43:19.289729 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:19.289697    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 26 11:43:24.296113 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:24.296083    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:43:24.296936 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:24.296883    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 26 11:43:25.315467 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:25.315433    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" event={"ID":"e9dff39d-f2cb-48e9-a3c1-3753107c084c","Type":"ContainerStarted","Data":"e8772bc7318bae070a5a142d1de2c76c595dd7769e367bda5b63fe9c8d812908"}
May 26 11:43:25.315831 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:25.315474    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" event={"ID":"e9dff39d-f2cb-48e9-a3c1-3753107c084c","Type":"ContainerStarted","Data":"334b614dc46f48844de13256e94a7f3e50010492dbf1bf3db4aec7a1effcf07d"}
May 26 11:43:25.315831 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:25.315682    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:25.338237 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:25.338177    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" podStartSLOduration=3.284344749 podStartE2EDuration="22.33815909s" podCreationTimestamp="2026-05-26 11:43:03 +0000 UTC" firstStartedPulling="2026-05-26 11:43:05.863462541 +0000 UTC m=+1118.669300998" lastFinishedPulling="2026-05-26 11:43:24.917276888 +0000 UTC m=+1137.723115339" observedRunningTime="2026-05-26 11:43:25.334933275 +0000 UTC m=+1138.140771739" watchObservedRunningTime="2026-05-26 11:43:25.33815909 +0000 UTC m=+1138.143997556"
May 26 11:43:26.319619 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:26.319577    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:26.321055 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:26.321017    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
May 26 11:43:27.323238 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:27.323198    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
May 26 11:43:32.327699 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:32.327669    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:43:32.328314 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:32.328261    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
May 26 11:43:34.297717 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:34.297676    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 26 11:43:42.329059 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:42.329020    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
May 26 11:43:44.297299 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:44.297240    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 26 11:43:52.329130 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:52.329090    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
May 26 11:43:54.296920 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:43:54.296883    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 26 11:44:02.328745 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:02.328704    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
May 26 11:44:04.297499 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:04.297468    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:44:12.328987 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:12.328946    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
May 26 11:44:22.328335 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:22.328260    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
May 26 11:44:32.329470 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:32.329437    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:44:33.013098 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.013063    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"]
May 26 11:44:33.013421 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.013374    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerName="kserve-container" containerID="cri-o://591fbd77eda6419fe38d8a1577b0d1af109e312b01a04438d133559765a4731a" gracePeriod=30
May 26 11:44:33.013544 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.013419    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerName="kube-rbac-proxy" containerID="cri-o://e24329d53edb13db0120e0f393022bb7a3b5798467edb9ca65b0be9857c26530" gracePeriod=30
May 26 11:44:33.082734 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.082699    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"]
May 26 11:44:33.086586 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.086564    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:44:33.089465 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.089443    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-90450-predictor-serving-cert\""
May 26 11:44:33.089465 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.089453    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-90450-kube-rbac-proxy-sar-config\""
May 26 11:44:33.101441 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.101415    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"]
May 26 11:44:33.146968 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.146943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl6x6\" (UniqueName: \"kubernetes.io/projected/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-kube-api-access-wl6x6\") pod \"success-200-isvc-90450-predictor-6d6789cd9d-flxmx\" (UID: \"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5\") " pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:44:33.147112 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.146987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-proxy-tls\") pod \"success-200-isvc-90450-predictor-6d6789cd9d-flxmx\" (UID: \"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5\") " pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:44:33.147160 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.147115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-90450-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-success-200-isvc-90450-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-90450-predictor-6d6789cd9d-flxmx\" (UID: \"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5\") " pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:44:33.247871 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.247828    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wl6x6\" (UniqueName: \"kubernetes.io/projected/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-kube-api-access-wl6x6\") pod \"success-200-isvc-90450-predictor-6d6789cd9d-flxmx\" (UID: \"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5\") " pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:44:33.247871 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.247874    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-proxy-tls\") pod \"success-200-isvc-90450-predictor-6d6789cd9d-flxmx\" (UID: \"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5\") " pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:44:33.248125 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.247932    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-90450-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-success-200-isvc-90450-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-90450-predictor-6d6789cd9d-flxmx\" (UID: \"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5\") " pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:44:33.248125 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:44:33.248049    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-90450-predictor-serving-cert: secret "success-200-isvc-90450-predictor-serving-cert" not found
May 26 11:44:33.248303 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:44:33.248133    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-proxy-tls podName:538ee6b7-39e3-44a1-a6aa-ce07fca82bc5 nodeName:}" failed. No retries permitted until 2026-05-26 11:44:33.748112107 +0000 UTC m=+1206.553950567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-proxy-tls") pod "success-200-isvc-90450-predictor-6d6789cd9d-flxmx" (UID: "538ee6b7-39e3-44a1-a6aa-ce07fca82bc5") : secret "success-200-isvc-90450-predictor-serving-cert" not found
May 26 11:44:33.248604 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.248585    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-90450-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-success-200-isvc-90450-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-90450-predictor-6d6789cd9d-flxmx\" (UID: \"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5\") " pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:44:33.258886 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.258850    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl6x6\" (UniqueName: \"kubernetes.io/projected/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-kube-api-access-wl6x6\") pod \"success-200-isvc-90450-predictor-6d6789cd9d-flxmx\" (UID: \"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5\") " pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:44:33.559267 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.559234    2563 generic.go:358] "Generic (PLEG): container finished" podID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerID="e24329d53edb13db0120e0f393022bb7a3b5798467edb9ca65b0be9857c26530" exitCode=2
May 26 11:44:33.559615 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.559306    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" event={"ID":"8c56111a-f26e-4266-bb10-86bdcdf8f2d7","Type":"ContainerDied","Data":"e24329d53edb13db0120e0f393022bb7a3b5798467edb9ca65b0be9857c26530"}
May 26 11:44:33.753124 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.753082    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-proxy-tls\") pod \"success-200-isvc-90450-predictor-6d6789cd9d-flxmx\" (UID: \"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5\") " pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:44:33.755543 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.755517    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-proxy-tls\") pod \"success-200-isvc-90450-predictor-6d6789cd9d-flxmx\" (UID: \"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5\") " pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:44:33.997835 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:33.997804    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:44:34.149442 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:34.149215    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"]
May 26 11:44:34.152068 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:44:34.152036    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538ee6b7_39e3_44a1_a6aa_ce07fca82bc5.slice/crio-bb5ab96bdb675e746f78dcb8f9c5c1406aa0e55cb16570c7d5db486183ee691c WatchSource:0}: Error finding container bb5ab96bdb675e746f78dcb8f9c5c1406aa0e55cb16570c7d5db486183ee691c: Status 404 returned error can't find the container with id bb5ab96bdb675e746f78dcb8f9c5c1406aa0e55cb16570c7d5db486183ee691c
May 26 11:44:34.290107 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:34.290074    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
May 26 11:44:34.297916 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:34.297887    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 26 11:44:34.565178 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:34.565098    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx" event={"ID":"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5","Type":"ContainerStarted","Data":"6b612b8c1588504945018793ba6a45df94569dfd484b729f574e3835045966ba"}
May 26 11:44:34.565178 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:34.565137    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx" event={"ID":"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5","Type":"ContainerStarted","Data":"b905753c082d59d5ea817bb51350666f476969334ed50b55a81da7c1889f4bc2"}
May 26 11:44:34.565178 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:34.565147    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx" event={"ID":"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5","Type":"ContainerStarted","Data":"bb5ab96bdb675e746f78dcb8f9c5c1406aa0e55cb16570c7d5db486183ee691c"}
May 26 11:44:34.565613 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:34.565236    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:44:34.585385 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:34.585335    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx" podStartSLOduration=1.5853218629999999 podStartE2EDuration="1.585321863s" podCreationTimestamp="2026-05-26 11:44:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:44:34.583844385 +0000 UTC m=+1207.389682851" watchObservedRunningTime="2026-05-26 11:44:34.585321863 +0000 UTC m=+1207.391160327"
May 26 11:44:35.569490 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:35.569457    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:44:35.570822 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:35.570793    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx" podUID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 11:44:36.264496 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.264472    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:44:36.376774 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.376684    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtbmd\" (UniqueName: \"kubernetes.io/projected/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-kube-api-access-dtbmd\") pod \"8c56111a-f26e-4266-bb10-86bdcdf8f2d7\" (UID: \"8c56111a-f26e-4266-bb10-86bdcdf8f2d7\") "
May 26 11:44:36.376774 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.376762    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-378f2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-success-200-isvc-378f2-kube-rbac-proxy-sar-config\") pod \"8c56111a-f26e-4266-bb10-86bdcdf8f2d7\" (UID: \"8c56111a-f26e-4266-bb10-86bdcdf8f2d7\") "
May 26 11:44:36.376989 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.376864    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-proxy-tls\") pod \"8c56111a-f26e-4266-bb10-86bdcdf8f2d7\" (UID: \"8c56111a-f26e-4266-bb10-86bdcdf8f2d7\") "
May 26 11:44:36.377066 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.377047    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-success-200-isvc-378f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-378f2-kube-rbac-proxy-sar-config") pod "8c56111a-f26e-4266-bb10-86bdcdf8f2d7" (UID: "8c56111a-f26e-4266-bb10-86bdcdf8f2d7"). InnerVolumeSpecName "success-200-isvc-378f2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:44:36.377133 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.377111    2563 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-378f2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-success-200-isvc-378f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:44:36.378927 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.378905    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-kube-api-access-dtbmd" (OuterVolumeSpecName: "kube-api-access-dtbmd") pod "8c56111a-f26e-4266-bb10-86bdcdf8f2d7" (UID: "8c56111a-f26e-4266-bb10-86bdcdf8f2d7"). InnerVolumeSpecName "kube-api-access-dtbmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:44:36.379042 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.379010    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c56111a-f26e-4266-bb10-86bdcdf8f2d7" (UID: "8c56111a-f26e-4266-bb10-86bdcdf8f2d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:44:36.478276 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.478247    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-proxy-tls\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:44:36.478276 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.478273    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtbmd\" (UniqueName: \"kubernetes.io/projected/8c56111a-f26e-4266-bb10-86bdcdf8f2d7-kube-api-access-dtbmd\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:44:36.573555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.573522    2563 generic.go:358] "Generic (PLEG): container finished" podID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerID="591fbd77eda6419fe38d8a1577b0d1af109e312b01a04438d133559765a4731a" exitCode=0
May 26 11:44:36.574022 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.573600    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" event={"ID":"8c56111a-f26e-4266-bb10-86bdcdf8f2d7","Type":"ContainerDied","Data":"591fbd77eda6419fe38d8a1577b0d1af109e312b01a04438d133559765a4731a"}
May 26 11:44:36.574022 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.573638    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd" event={"ID":"8c56111a-f26e-4266-bb10-86bdcdf8f2d7","Type":"ContainerDied","Data":"67154915a93b5b5b8d545d15a907ae50cab266d4c009edf357739e8a596cd336"}
May 26 11:44:36.574022 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.573653    2563 scope.go:117] "RemoveContainer" containerID="e24329d53edb13db0120e0f393022bb7a3b5798467edb9ca65b0be9857c26530"
May 26 11:44:36.574022 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.573611    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"
May 26 11:44:36.574476 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.574448    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx" podUID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 11:44:36.583059 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.583044    2563 scope.go:117] "RemoveContainer" containerID="591fbd77eda6419fe38d8a1577b0d1af109e312b01a04438d133559765a4731a"
May 26 11:44:36.590757 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.590736    2563 scope.go:117] "RemoveContainer" containerID="e24329d53edb13db0120e0f393022bb7a3b5798467edb9ca65b0be9857c26530"
May 26 11:44:36.591011 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:44:36.590990    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24329d53edb13db0120e0f393022bb7a3b5798467edb9ca65b0be9857c26530\": container with ID starting with e24329d53edb13db0120e0f393022bb7a3b5798467edb9ca65b0be9857c26530 not found: ID does not exist" containerID="e24329d53edb13db0120e0f393022bb7a3b5798467edb9ca65b0be9857c26530"
May 26 11:44:36.591075 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.591019    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24329d53edb13db0120e0f393022bb7a3b5798467edb9ca65b0be9857c26530"} err="failed to get container status \"e24329d53edb13db0120e0f393022bb7a3b5798467edb9ca65b0be9857c26530\": rpc error: code = NotFound desc = could not find container \"e24329d53edb13db0120e0f393022bb7a3b5798467edb9ca65b0be9857c26530\": container with ID starting with e24329d53edb13db0120e0f393022bb7a3b5798467edb9ca65b0be9857c26530 not found: ID does not exist"
May 26 11:44:36.591075 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.591038    2563 scope.go:117] "RemoveContainer" containerID="591fbd77eda6419fe38d8a1577b0d1af109e312b01a04438d133559765a4731a"
May 26 11:44:36.591314 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:44:36.591275    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"591fbd77eda6419fe38d8a1577b0d1af109e312b01a04438d133559765a4731a\": container with ID starting with 591fbd77eda6419fe38d8a1577b0d1af109e312b01a04438d133559765a4731a not found: ID does not exist" containerID="591fbd77eda6419fe38d8a1577b0d1af109e312b01a04438d133559765a4731a"
May 26 11:44:36.591360 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.591322    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"591fbd77eda6419fe38d8a1577b0d1af109e312b01a04438d133559765a4731a"} err="failed to get container status \"591fbd77eda6419fe38d8a1577b0d1af109e312b01a04438d133559765a4731a\": rpc error: code = NotFound desc = could not find container \"591fbd77eda6419fe38d8a1577b0d1af109e312b01a04438d133559765a4731a\": container with ID starting with 591fbd77eda6419fe38d8a1577b0d1af109e312b01a04438d133559765a4731a not found: ID does not exist"
May 26 11:44:36.596777 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.596755    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"]
May 26 11:44:36.600231 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:36.600211    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-378f2-predictor-8bbd48d54-vp5rd"]
May 26 11:44:37.797865 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:37.797831    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" path="/var/lib/kubelet/pods/8c56111a-f26e-4266-bb10-86bdcdf8f2d7/volumes"
May 26 11:44:41.578015 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:41.577981    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:44:41.578578 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:41.578549    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx" podUID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 11:44:51.579045 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:44:51.579000    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx" podUID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 11:45:01.579373 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:01.579319    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx" podUID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 11:45:02.973569 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:02.973531    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"]
May 26 11:45:02.974063 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:02.974007    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kserve-container" containerID="cri-o://334b614dc46f48844de13256e94a7f3e50010492dbf1bf3db4aec7a1effcf07d" gracePeriod=30
May 26 11:45:02.974200 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:02.974178    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kube-rbac-proxy" containerID="cri-o://e8772bc7318bae070a5a142d1de2c76c595dd7769e367bda5b63fe9c8d812908" gracePeriod=30
May 26 11:45:02.994878 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:02.994850    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"]
May 26 11:45:02.995491 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:02.995474    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerName="kserve-container"
May 26 11:45:02.995491 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:02.995492    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerName="kserve-container"
May 26 11:45:02.995648 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:02.995511    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerName="kube-rbac-proxy"
May 26 11:45:02.995648 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:02.995517    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerName="kube-rbac-proxy"
May 26 11:45:02.995648 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:02.995569    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerName="kube-rbac-proxy"
May 26 11:45:02.995648 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:02.995578    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="8c56111a-f26e-4266-bb10-86bdcdf8f2d7" containerName="kserve-container"
May 26 11:45:02.998451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:02.998430    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:45:03.000717 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:03.000677    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5b0de-predictor-serving-cert\""
May 26 11:45:03.001013 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:03.000920    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5b0de-kube-rbac-proxy-sar-config\""
May 26 11:45:03.007780 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:03.006924    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"]
May 26 11:45:03.104023 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:03.103974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-proxy-tls\") pod \"success-200-isvc-5b0de-predictor-7945cf7494-zdzng\" (UID: \"31cac73a-d9f6-4a14-b4fc-d090af94f5d7\") " pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:45:03.104409 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:03.104353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-5b0de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-success-200-isvc-5b0de-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5b0de-predictor-7945cf7494-zdzng\" (UID: \"31cac73a-d9f6-4a14-b4fc-d090af94f5d7\") " pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:45:03.104409 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:03.104400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd9c2\" (UniqueName: \"kubernetes.io/projected/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-kube-api-access-kd9c2\") pod \"success-200-isvc-5b0de-predictor-7945cf7494-zdzng\" (UID: \"31cac73a-d9f6-4a14-b4fc-d090af94f5d7\") " pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:45:03.205438 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:03.205405    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-5b0de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-success-200-isvc-5b0de-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5b0de-predictor-7945cf7494-zdzng\" (UID: \"31cac73a-d9f6-4a14-b4fc-d090af94f5d7\") " pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:45:03.205438 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:03.205441    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9c2\" (UniqueName: \"kubernetes.io/projected/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-kube-api-access-kd9c2\") pod \"success-200-isvc-5b0de-predictor-7945cf7494-zdzng\" (UID: \"31cac73a-d9f6-4a14-b4fc-d090af94f5d7\") " pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:45:03.205695 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:03.205469    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-proxy-tls\") pod \"success-200-isvc-5b0de-predictor-7945cf7494-zdzng\" (UID: \"31cac73a-d9f6-4a14-b4fc-d090af94f5d7\") " pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:45:03.205695 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:45:03.205633    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-serving-cert: secret "success-200-isvc-5b0de-predictor-serving-cert" not found
May 26 11:45:03.205814 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:45:03.205709    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-proxy-tls podName:31cac73a-d9f6-4a14-b4fc-d090af94f5d7 nodeName:}" failed. No retries permitted until 2026-05-26 11:45:03.705687812 +0000 UTC m=+1236.511526262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-proxy-tls") pod "success-200-isvc-5b0de-predictor-7945cf7494-zdzng" (UID: "31cac73a-d9f6-4a14-b4fc-d090af94f5d7") : secret "success-200-isvc-5b0de-predictor-serving-cert" not found
May 26 11:45:03.206136 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:03.206107    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-5b0de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-success-200-isvc-5b0de-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5b0de-predictor-7945cf7494-zdzng\" (UID: \"31cac73a-d9f6-4a14-b4fc-d090af94f5d7\") " pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:45:03.214119 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:03.214093    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd9c2\" (UniqueName: \"kubernetes.io/projected/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-kube-api-access-kd9c2\") pod \"success-200-isvc-5b0de-predictor-7945cf7494-zdzng\" (UID: \"31cac73a-d9f6-4a14-b4fc-d090af94f5d7\") " pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:45:03.670232 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:03.670197    2563 generic.go:358] "Generic (PLEG): container finished" podID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerID="e8772bc7318bae070a5a142d1de2c76c595dd7769e367bda5b63fe9c8d812908" exitCode=2
May 26 11:45:03.670432 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:03.670267    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" event={"ID":"e9dff39d-f2cb-48e9-a3c1-3753107c084c","Type":"ContainerDied","Data":"e8772bc7318bae070a5a142d1de2c76c595dd7769e367bda5b63fe9c8d812908"}
May 26 11:45:03.709383 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:03.709344    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-proxy-tls\") pod \"success-200-isvc-5b0de-predictor-7945cf7494-zdzng\" (UID: \"31cac73a-d9f6-4a14-b4fc-d090af94f5d7\") " pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:45:03.711780 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:03.711759    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-proxy-tls\") pod \"success-200-isvc-5b0de-predictor-7945cf7494-zdzng\" (UID: \"31cac73a-d9f6-4a14-b4fc-d090af94f5d7\") " pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:45:03.911914 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:03.911879    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:45:04.040382 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:04.040270    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"]
May 26 11:45:04.042957 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:45:04.042931    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31cac73a_d9f6_4a14_b4fc_d090af94f5d7.slice/crio-e07ed1390605925fea8e6b446d66ecf628ef5b358c4dc6b570be7a986b354934 WatchSource:0}: Error finding container e07ed1390605925fea8e6b446d66ecf628ef5b358c4dc6b570be7a986b354934: Status 404 returned error can't find the container with id e07ed1390605925fea8e6b446d66ecf628ef5b358c4dc6b570be7a986b354934
May 26 11:45:04.044783 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:04.044761    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:45:04.674992 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:04.674950    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng" event={"ID":"31cac73a-d9f6-4a14-b4fc-d090af94f5d7","Type":"ContainerStarted","Data":"a14861bdef22b6e403f78930fe06a36e97097c544f656ed5b16e2691e5094f33"}
May 26 11:45:04.674992 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:04.674995    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng" event={"ID":"31cac73a-d9f6-4a14-b4fc-d090af94f5d7","Type":"ContainerStarted","Data":"bf845ebfdb52f31bca755abe985ac257af308297653deced6d5a8e89a0ce8688"}
May 26 11:45:04.675206 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:04.675009    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng" event={"ID":"31cac73a-d9f6-4a14-b4fc-d090af94f5d7","Type":"ContainerStarted","Data":"e07ed1390605925fea8e6b446d66ecf628ef5b358c4dc6b570be7a986b354934"}
May 26 11:45:04.675206 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:04.675104    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:45:04.694556 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:04.694507    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng" podStartSLOduration=2.694493746 podStartE2EDuration="2.694493746s" podCreationTimestamp="2026-05-26 11:45:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:45:04.692721639 +0000 UTC m=+1237.498560125" watchObservedRunningTime="2026-05-26 11:45:04.694493746 +0000 UTC m=+1237.500332211"
May 26 11:45:05.679078 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:05.679033    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:45:05.680168 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:05.680133    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng" podUID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 26 11:45:06.681737 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:06.681685    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng" podUID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 26 11:45:07.323986 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.323945    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
May 26 11:45:07.622200 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.622175    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:45:07.686509 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.686474    2563 generic.go:358] "Generic (PLEG): container finished" podID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerID="334b614dc46f48844de13256e94a7f3e50010492dbf1bf3db4aec7a1effcf07d" exitCode=0
May 26 11:45:07.686926 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.686540    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" event={"ID":"e9dff39d-f2cb-48e9-a3c1-3753107c084c","Type":"ContainerDied","Data":"334b614dc46f48844de13256e94a7f3e50010492dbf1bf3db4aec7a1effcf07d"}
May 26 11:45:07.686926 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.686579    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p" event={"ID":"e9dff39d-f2cb-48e9-a3c1-3753107c084c","Type":"ContainerDied","Data":"da7172eaae3d145fa883b03691b0a1b4e7d2d492ad0f44bfe458146985bb4a72"}
May 26 11:45:07.686926 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.686590    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"
May 26 11:45:07.686926 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.686601    2563 scope.go:117] "RemoveContainer" containerID="e8772bc7318bae070a5a142d1de2c76c595dd7769e367bda5b63fe9c8d812908"
May 26 11:45:07.694971 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.694946    2563 scope.go:117] "RemoveContainer" containerID="334b614dc46f48844de13256e94a7f3e50010492dbf1bf3db4aec7a1effcf07d"
May 26 11:45:07.702471 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.702453    2563 scope.go:117] "RemoveContainer" containerID="8bdfadfce1d73c9eab5ac1e8547786a4153463d709dd1e4b95639e62d165f5d5"
May 26 11:45:07.709577 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.709558    2563 scope.go:117] "RemoveContainer" containerID="e8772bc7318bae070a5a142d1de2c76c595dd7769e367bda5b63fe9c8d812908"
May 26 11:45:07.709810 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:45:07.709789    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8772bc7318bae070a5a142d1de2c76c595dd7769e367bda5b63fe9c8d812908\": container with ID starting with e8772bc7318bae070a5a142d1de2c76c595dd7769e367bda5b63fe9c8d812908 not found: ID does not exist" containerID="e8772bc7318bae070a5a142d1de2c76c595dd7769e367bda5b63fe9c8d812908"
May 26 11:45:07.709856 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.709820    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8772bc7318bae070a5a142d1de2c76c595dd7769e367bda5b63fe9c8d812908"} err="failed to get container status \"e8772bc7318bae070a5a142d1de2c76c595dd7769e367bda5b63fe9c8d812908\": rpc error: code = NotFound desc = could not find container \"e8772bc7318bae070a5a142d1de2c76c595dd7769e367bda5b63fe9c8d812908\": container with ID starting with e8772bc7318bae070a5a142d1de2c76c595dd7769e367bda5b63fe9c8d812908 not found: ID does not exist"
May 26 11:45:07.709856 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.709840    2563 scope.go:117] "RemoveContainer" containerID="334b614dc46f48844de13256e94a7f3e50010492dbf1bf3db4aec7a1effcf07d"
May 26 11:45:07.710062 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:45:07.710045    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334b614dc46f48844de13256e94a7f3e50010492dbf1bf3db4aec7a1effcf07d\": container with ID starting with 334b614dc46f48844de13256e94a7f3e50010492dbf1bf3db4aec7a1effcf07d not found: ID does not exist" containerID="334b614dc46f48844de13256e94a7f3e50010492dbf1bf3db4aec7a1effcf07d"
May 26 11:45:07.710131 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.710084    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334b614dc46f48844de13256e94a7f3e50010492dbf1bf3db4aec7a1effcf07d"} err="failed to get container status \"334b614dc46f48844de13256e94a7f3e50010492dbf1bf3db4aec7a1effcf07d\": rpc error: code = NotFound desc = could not find container \"334b614dc46f48844de13256e94a7f3e50010492dbf1bf3db4aec7a1effcf07d\": container with ID starting with 334b614dc46f48844de13256e94a7f3e50010492dbf1bf3db4aec7a1effcf07d not found: ID does not exist"
May 26 11:45:07.710131 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.710108    2563 scope.go:117] "RemoveContainer" containerID="8bdfadfce1d73c9eab5ac1e8547786a4153463d709dd1e4b95639e62d165f5d5"
May 26 11:45:07.710373 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:45:07.710356    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bdfadfce1d73c9eab5ac1e8547786a4153463d709dd1e4b95639e62d165f5d5\": container with ID starting with 8bdfadfce1d73c9eab5ac1e8547786a4153463d709dd1e4b95639e62d165f5d5 not found: ID does not exist" containerID="8bdfadfce1d73c9eab5ac1e8547786a4153463d709dd1e4b95639e62d165f5d5"
May 26 11:45:07.710423 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.710377    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bdfadfce1d73c9eab5ac1e8547786a4153463d709dd1e4b95639e62d165f5d5"} err="failed to get container status \"8bdfadfce1d73c9eab5ac1e8547786a4153463d709dd1e4b95639e62d165f5d5\": rpc error: code = NotFound desc = could not find container \"8bdfadfce1d73c9eab5ac1e8547786a4153463d709dd1e4b95639e62d165f5d5\": container with ID starting with 8bdfadfce1d73c9eab5ac1e8547786a4153463d709dd1e4b95639e62d165f5d5 not found: ID does not exist"
May 26 11:45:07.748425 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.748392    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hkqb\" (UniqueName: \"kubernetes.io/projected/e9dff39d-f2cb-48e9-a3c1-3753107c084c-kube-api-access-8hkqb\") pod \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") "
May 26 11:45:07.748586 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.748449    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9dff39d-f2cb-48e9-a3c1-3753107c084c-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") "
May 26 11:45:07.748586 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.748480    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9dff39d-f2cb-48e9-a3c1-3753107c084c-proxy-tls\") pod \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") "
May 26 11:45:07.748586 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.748531    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9dff39d-f2cb-48e9-a3c1-3753107c084c-kserve-provision-location\") pod \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\" (UID: \"e9dff39d-f2cb-48e9-a3c1-3753107c084c\") "
May 26 11:45:07.748933 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.748899    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9dff39d-f2cb-48e9-a3c1-3753107c084c-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-2-kube-rbac-proxy-sar-config") pod "e9dff39d-f2cb-48e9-a3c1-3753107c084c" (UID: "e9dff39d-f2cb-48e9-a3c1-3753107c084c"). InnerVolumeSpecName "isvc-sklearn-graph-2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:45:07.749009 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.748911    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9dff39d-f2cb-48e9-a3c1-3753107c084c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9dff39d-f2cb-48e9-a3c1-3753107c084c" (UID: "e9dff39d-f2cb-48e9-a3c1-3753107c084c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:45:07.750574 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.750552    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9dff39d-f2cb-48e9-a3c1-3753107c084c-kube-api-access-8hkqb" (OuterVolumeSpecName: "kube-api-access-8hkqb") pod "e9dff39d-f2cb-48e9-a3c1-3753107c084c" (UID: "e9dff39d-f2cb-48e9-a3c1-3753107c084c"). InnerVolumeSpecName "kube-api-access-8hkqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:45:07.750649 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.750595    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9dff39d-f2cb-48e9-a3c1-3753107c084c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e9dff39d-f2cb-48e9-a3c1-3753107c084c" (UID: "e9dff39d-f2cb-48e9-a3c1-3753107c084c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:45:07.850250 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.850213    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9dff39d-f2cb-48e9-a3c1-3753107c084c-proxy-tls\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:45:07.850250 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.850244    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9dff39d-f2cb-48e9-a3c1-3753107c084c-kserve-provision-location\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:45:07.850458 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.850257    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hkqb\" (UniqueName: \"kubernetes.io/projected/e9dff39d-f2cb-48e9-a3c1-3753107c084c-kube-api-access-8hkqb\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:45:07.850458 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:07.850272    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9dff39d-f2cb-48e9-a3c1-3753107c084c-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:45:08.003063 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:08.003033    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"]
May 26 11:45:08.007788 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:08.007762    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-fjv9p"]
May 26 11:45:09.796614 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:09.796584    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" path="/var/lib/kubelet/pods/e9dff39d-f2cb-48e9-a3c1-3753107c084c/volumes"
May 26 11:45:11.579162 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:11.579119    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx" podUID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 11:45:11.686802 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:11.686773    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:45:11.687223 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:11.687197    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng" podUID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 26 11:45:21.579262 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:21.579230    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:45:21.688025 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:21.687987    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng" podUID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 26 11:45:31.688179 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:31.688130    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng" podUID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 26 11:45:41.687556 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:41.687504    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng" podUID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 26 11:45:51.687943 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:45:51.687912    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:53:47.988758 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:47.988659    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"]
May 26 11:53:47.988758 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:47.989088    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx" podUID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" containerName="kserve-container" containerID="cri-o://b905753c082d59d5ea817bb51350666f476969334ed50b55a81da7c1889f4bc2" gracePeriod=30
May 26 11:53:47.991713 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:47.989140    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx" podUID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" containerName="kube-rbac-proxy" containerID="cri-o://6b612b8c1588504945018793ba6a45df94569dfd484b729f574e3835045966ba" gracePeriod=30
May 26 11:53:48.061138 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.061104    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"]
May 26 11:53:48.061958 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.061939    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kserve-container"
May 26 11:53:48.062096 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.062076    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kserve-container"
May 26 11:53:48.062223 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.062116    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="storage-initializer"
May 26 11:53:48.062223 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.062126    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="storage-initializer"
May 26 11:53:48.062223 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.062139    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kube-rbac-proxy"
May 26 11:53:48.062223 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.062147    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kube-rbac-proxy"
May 26 11:53:48.062411 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.062241    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kube-rbac-proxy"
May 26 11:53:48.062411 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.062256    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e9dff39d-f2cb-48e9-a3c1-3753107c084c" containerName="kserve-container"
May 26 11:53:48.065613 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.065595    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:53:48.067742 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.067722    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5380a-kube-rbac-proxy-sar-config\""
May 26 11:53:48.067917 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.067899    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5380a-predictor-serving-cert\""
May 26 11:53:48.085195 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.085162    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"]
May 26 11:53:48.196785 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.196747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-5380a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-success-200-isvc-5380a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw\" (UID: \"56b04d4a-e2b3-43aa-a6f1-56035cdb156d\") " pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:53:48.196785 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.196791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58wvw\" (UniqueName: \"kubernetes.io/projected/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-kube-api-access-58wvw\") pod \"success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw\" (UID: \"56b04d4a-e2b3-43aa-a6f1-56035cdb156d\") " pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:53:48.197001 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.196846    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-proxy-tls\") pod \"success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw\" (UID: \"56b04d4a-e2b3-43aa-a6f1-56035cdb156d\") " pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:53:48.297623 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.297543    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-5380a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-success-200-isvc-5380a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw\" (UID: \"56b04d4a-e2b3-43aa-a6f1-56035cdb156d\") " pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:53:48.297623 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.297588    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-58wvw\" (UniqueName: \"kubernetes.io/projected/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-kube-api-access-58wvw\") pod \"success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw\" (UID: \"56b04d4a-e2b3-43aa-a6f1-56035cdb156d\") " pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:53:48.297623 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.297624    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-proxy-tls\") pod \"success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw\" (UID: \"56b04d4a-e2b3-43aa-a6f1-56035cdb156d\") " pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:53:48.298201 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.298171    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-5380a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-success-200-isvc-5380a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw\" (UID: \"56b04d4a-e2b3-43aa-a6f1-56035cdb156d\") " pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:53:48.299991 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.299970    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-proxy-tls\") pod \"success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw\" (UID: \"56b04d4a-e2b3-43aa-a6f1-56035cdb156d\") " pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:53:48.306371 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.306347    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58wvw\" (UniqueName: \"kubernetes.io/projected/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-kube-api-access-58wvw\") pod \"success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw\" (UID: \"56b04d4a-e2b3-43aa-a6f1-56035cdb156d\") " pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:53:48.376303 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.376260    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:53:48.485856 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.485820    2563 generic.go:358] "Generic (PLEG): container finished" podID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" containerID="6b612b8c1588504945018793ba6a45df94569dfd484b729f574e3835045966ba" exitCode=2
May 26 11:53:48.486032 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.485879    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx" event={"ID":"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5","Type":"ContainerDied","Data":"6b612b8c1588504945018793ba6a45df94569dfd484b729f574e3835045966ba"}
May 26 11:53:48.511836 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.511690    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"]
May 26 11:53:48.515792 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:53:48.515765    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b04d4a_e2b3_43aa_a6f1_56035cdb156d.slice/crio-a48cb8aba3ca424dbd047de944dc41936040977fff40b69d45f9dd23fa7c7e0b WatchSource:0}: Error finding container a48cb8aba3ca424dbd047de944dc41936040977fff40b69d45f9dd23fa7c7e0b: Status 404 returned error can't find the container with id a48cb8aba3ca424dbd047de944dc41936040977fff40b69d45f9dd23fa7c7e0b
May 26 11:53:48.519669 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:48.519649    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:53:49.490605 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:49.490569    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw" event={"ID":"56b04d4a-e2b3-43aa-a6f1-56035cdb156d","Type":"ContainerStarted","Data":"61cd67aaeb3e735e93871b6126dc84dafbb9ba517671d9bb71f6e7e24afdbf39"}
May 26 11:53:49.490605 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:49.490610    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw" event={"ID":"56b04d4a-e2b3-43aa-a6f1-56035cdb156d","Type":"ContainerStarted","Data":"fa85650fe6b62aeac70fbe87fba116dace0badedc7998e525e1c8893d9ad80fd"}
May 26 11:53:49.491030 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:49.490622    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw" event={"ID":"56b04d4a-e2b3-43aa-a6f1-56035cdb156d","Type":"ContainerStarted","Data":"a48cb8aba3ca424dbd047de944dc41936040977fff40b69d45f9dd23fa7c7e0b"}
May 26 11:53:49.491030 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:49.490710    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:53:49.510564 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:49.510511    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw" podStartSLOduration=1.5104977769999999 podStartE2EDuration="1.510497777s" podCreationTimestamp="2026-05-26 11:53:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:53:49.50954789 +0000 UTC m=+1762.315386353" watchObservedRunningTime="2026-05-26 11:53:49.510497777 +0000 UTC m=+1762.316336241"
May 26 11:53:50.495148 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:50.495109    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:53:50.496263 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:50.496235    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw" podUID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 26 11:53:51.238174 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.238147    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:53:51.323003 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.322907    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl6x6\" (UniqueName: \"kubernetes.io/projected/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-kube-api-access-wl6x6\") pod \"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5\" (UID: \"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5\") "
May 26 11:53:51.323182 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.323038    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-90450-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-success-200-isvc-90450-kube-rbac-proxy-sar-config\") pod \"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5\" (UID: \"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5\") "
May 26 11:53:51.323182 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.323140    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-proxy-tls\") pod \"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5\" (UID: \"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5\") "
May 26 11:53:51.323402 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.323377    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-success-200-isvc-90450-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-90450-kube-rbac-proxy-sar-config") pod "538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" (UID: "538ee6b7-39e3-44a1-a6aa-ce07fca82bc5"). InnerVolumeSpecName "success-200-isvc-90450-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:53:51.325007 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.324982    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-kube-api-access-wl6x6" (OuterVolumeSpecName: "kube-api-access-wl6x6") pod "538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" (UID: "538ee6b7-39e3-44a1-a6aa-ce07fca82bc5"). InnerVolumeSpecName "kube-api-access-wl6x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:53:51.325114 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.325059    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" (UID: "538ee6b7-39e3-44a1-a6aa-ce07fca82bc5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:53:51.424401 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.424361    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl6x6\" (UniqueName: \"kubernetes.io/projected/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-kube-api-access-wl6x6\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:53:51.424401 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.424395    2563 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-90450-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-success-200-isvc-90450-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:53:51.424611 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.424413    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5-proxy-tls\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:53:51.499379 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.499348    2563 generic.go:358] "Generic (PLEG): container finished" podID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" containerID="b905753c082d59d5ea817bb51350666f476969334ed50b55a81da7c1889f4bc2" exitCode=0
May 26 11:53:51.499752 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.499429    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"
May 26 11:53:51.499752 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.499445    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx" event={"ID":"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5","Type":"ContainerDied","Data":"b905753c082d59d5ea817bb51350666f476969334ed50b55a81da7c1889f4bc2"}
May 26 11:53:51.499752 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.499498    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx" event={"ID":"538ee6b7-39e3-44a1-a6aa-ce07fca82bc5","Type":"ContainerDied","Data":"bb5ab96bdb675e746f78dcb8f9c5c1406aa0e55cb16570c7d5db486183ee691c"}
May 26 11:53:51.499752 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.499512    2563 scope.go:117] "RemoveContainer" containerID="6b612b8c1588504945018793ba6a45df94569dfd484b729f574e3835045966ba"
May 26 11:53:51.499959 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.499935    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw" podUID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 26 11:53:51.508645 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.508635    2563 scope.go:117] "RemoveContainer" containerID="b905753c082d59d5ea817bb51350666f476969334ed50b55a81da7c1889f4bc2"
May 26 11:53:51.515803 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.515787    2563 scope.go:117] "RemoveContainer" containerID="6b612b8c1588504945018793ba6a45df94569dfd484b729f574e3835045966ba"
May 26 11:53:51.516041 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:53:51.516023    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b612b8c1588504945018793ba6a45df94569dfd484b729f574e3835045966ba\": container with ID starting with 6b612b8c1588504945018793ba6a45df94569dfd484b729f574e3835045966ba not found: ID does not exist" containerID="6b612b8c1588504945018793ba6a45df94569dfd484b729f574e3835045966ba"
May 26 11:53:51.516105 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.516052    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b612b8c1588504945018793ba6a45df94569dfd484b729f574e3835045966ba"} err="failed to get container status \"6b612b8c1588504945018793ba6a45df94569dfd484b729f574e3835045966ba\": rpc error: code = NotFound desc = could not find container \"6b612b8c1588504945018793ba6a45df94569dfd484b729f574e3835045966ba\": container with ID starting with 6b612b8c1588504945018793ba6a45df94569dfd484b729f574e3835045966ba not found: ID does not exist"
May 26 11:53:51.516105 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.516076    2563 scope.go:117] "RemoveContainer" containerID="b905753c082d59d5ea817bb51350666f476969334ed50b55a81da7c1889f4bc2"
May 26 11:53:51.516330 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:53:51.516312    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b905753c082d59d5ea817bb51350666f476969334ed50b55a81da7c1889f4bc2\": container with ID starting with b905753c082d59d5ea817bb51350666f476969334ed50b55a81da7c1889f4bc2 not found: ID does not exist" containerID="b905753c082d59d5ea817bb51350666f476969334ed50b55a81da7c1889f4bc2"
May 26 11:53:51.516391 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.516337    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b905753c082d59d5ea817bb51350666f476969334ed50b55a81da7c1889f4bc2"} err="failed to get container status \"b905753c082d59d5ea817bb51350666f476969334ed50b55a81da7c1889f4bc2\": rpc error: code = NotFound desc = could not find container \"b905753c082d59d5ea817bb51350666f476969334ed50b55a81da7c1889f4bc2\": container with ID starting with b905753c082d59d5ea817bb51350666f476969334ed50b55a81da7c1889f4bc2 not found: ID does not exist"
May 26 11:53:51.521565 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.521543    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"]
May 26 11:53:51.525501 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.525477    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-90450-predictor-6d6789cd9d-flxmx"]
May 26 11:53:51.797155 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:51.797124    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" path="/var/lib/kubelet/pods/538ee6b7-39e3-44a1-a6aa-ce07fca82bc5/volumes"
May 26 11:53:56.504623 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:56.504591    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:53:56.505181 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:53:56.505155    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw" podUID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 26 11:54:06.505461 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:06.505419    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw" podUID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 26 11:54:16.505560 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:16.505519    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw" podUID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 26 11:54:17.900791 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.900745    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"]
May 26 11:54:17.901214 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.901099    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng" podUID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" containerName="kserve-container" containerID="cri-o://bf845ebfdb52f31bca755abe985ac257af308297653deced6d5a8e89a0ce8688" gracePeriod=30
May 26 11:54:17.901323 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.901194    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng" podUID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" containerName="kube-rbac-proxy" containerID="cri-o://a14861bdef22b6e403f78930fe06a36e97097c544f656ed5b16e2691e5094f33" gracePeriod=30
May 26 11:54:17.936664 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.936630    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"]
May 26 11:54:17.937197 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.937177    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" containerName="kserve-container"
May 26 11:54:17.937197 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.937195    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" containerName="kserve-container"
May 26 11:54:17.937373 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.937215    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" containerName="kube-rbac-proxy"
May 26 11:54:17.937373 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.937223    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" containerName="kube-rbac-proxy"
May 26 11:54:17.937373 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.937347    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" containerName="kube-rbac-proxy"
May 26 11:54:17.937373 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.937360    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="538ee6b7-39e3-44a1-a6aa-ce07fca82bc5" containerName="kserve-container"
May 26 11:54:17.941066 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.941046    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:54:17.943663 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.943638    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fbdd2-kube-rbac-proxy-sar-config\""
May 26 11:54:17.943809 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.943638    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fbdd2-predictor-serving-cert\""
May 26 11:54:17.951390 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.951368    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"]
May 26 11:54:17.955510 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.955246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-fbdd2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-success-200-isvc-fbdd2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x\" (UID: \"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b\") " pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:54:17.955510 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.955357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-proxy-tls\") pod \"success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x\" (UID: \"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b\") " pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:54:17.955510 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:17.955391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldktr\" (UniqueName: \"kubernetes.io/projected/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-kube-api-access-ldktr\") pod \"success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x\" (UID: \"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b\") " pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:54:18.055851 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:18.055817    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-proxy-tls\") pod \"success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x\" (UID: \"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b\") " pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:54:18.056041 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:18.055859    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ldktr\" (UniqueName: \"kubernetes.io/projected/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-kube-api-access-ldktr\") pod \"success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x\" (UID: \"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b\") " pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:54:18.056041 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:18.055963    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-fbdd2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-success-200-isvc-fbdd2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x\" (UID: \"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b\") " pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:54:18.056041 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:54:18.056002    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-serving-cert: secret "success-200-isvc-fbdd2-predictor-serving-cert" not found
May 26 11:54:18.056219 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:54:18.056081    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-proxy-tls podName:6399324a-aa3d-4bd2-ae41-36df7c5ccb5b nodeName:}" failed. No retries permitted until 2026-05-26 11:54:18.556060746 +0000 UTC m=+1791.361899192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-proxy-tls") pod "success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x" (UID: "6399324a-aa3d-4bd2-ae41-36df7c5ccb5b") : secret "success-200-isvc-fbdd2-predictor-serving-cert" not found
May 26 11:54:18.056677 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:18.056648    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-fbdd2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-success-200-isvc-fbdd2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x\" (UID: \"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b\") " pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:54:18.066359 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:18.066327    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldktr\" (UniqueName: \"kubernetes.io/projected/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-kube-api-access-ldktr\") pod \"success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x\" (UID: \"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b\") " pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:54:18.559876 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:18.559840    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-proxy-tls\") pod \"success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x\" (UID: \"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b\") " pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:54:18.562359 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:18.562336    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-proxy-tls\") pod \"success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x\" (UID: \"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b\") " pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:54:18.595814 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:18.595777    2563 generic.go:358] "Generic (PLEG): container finished" podID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" containerID="a14861bdef22b6e403f78930fe06a36e97097c544f656ed5b16e2691e5094f33" exitCode=2
May 26 11:54:18.595976 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:18.595838    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng" event={"ID":"31cac73a-d9f6-4a14-b4fc-d090af94f5d7","Type":"ContainerDied","Data":"a14861bdef22b6e403f78930fe06a36e97097c544f656ed5b16e2691e5094f33"}
May 26 11:54:18.858221 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:18.858139    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:54:18.988901 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:18.988868    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"]
May 26 11:54:18.992452 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:54:18.992422    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6399324a_aa3d_4bd2_ae41_36df7c5ccb5b.slice/crio-e97eb01a78074a5612bc7f61378235094300293d80d9c54ec63397669db81442 WatchSource:0}: Error finding container e97eb01a78074a5612bc7f61378235094300293d80d9c54ec63397669db81442: Status 404 returned error can't find the container with id e97eb01a78074a5612bc7f61378235094300293d80d9c54ec63397669db81442
May 26 11:54:19.601387 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:19.601343    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x" event={"ID":"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b","Type":"ContainerStarted","Data":"28ea78423059c350e5c294275f521f22b1d4e252e6d00f4f826d73b39ea7b324"}
May 26 11:54:19.601387 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:19.601389    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x" event={"ID":"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b","Type":"ContainerStarted","Data":"e7e861287d27562041147fe752035c5da2acc7f39b0cc4c6a4f62d9b411cbff4"}
May 26 11:54:19.601687 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:19.601403    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x" event={"ID":"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b","Type":"ContainerStarted","Data":"e97eb01a78074a5612bc7f61378235094300293d80d9c54ec63397669db81442"}
May 26 11:54:19.601687 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:19.601498    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:54:19.620505 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:19.620458    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x" podStartSLOduration=2.620422318 podStartE2EDuration="2.620422318s" podCreationTimestamp="2026-05-26 11:54:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:54:19.618533435 +0000 UTC m=+1792.424371898" watchObservedRunningTime="2026-05-26 11:54:19.620422318 +0000 UTC m=+1792.426260782"
May 26 11:54:20.606017 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:20.605984    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:54:20.607313 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:20.607273    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x" podUID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 26 11:54:21.158001 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.157978    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:54:21.184775 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.184702    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-proxy-tls\") pod \"31cac73a-d9f6-4a14-b4fc-d090af94f5d7\" (UID: \"31cac73a-d9f6-4a14-b4fc-d090af94f5d7\") "
May 26 11:54:21.184916 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.184812    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-5b0de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-success-200-isvc-5b0de-kube-rbac-proxy-sar-config\") pod \"31cac73a-d9f6-4a14-b4fc-d090af94f5d7\" (UID: \"31cac73a-d9f6-4a14-b4fc-d090af94f5d7\") "
May 26 11:54:21.184916 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.184840    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd9c2\" (UniqueName: \"kubernetes.io/projected/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-kube-api-access-kd9c2\") pod \"31cac73a-d9f6-4a14-b4fc-d090af94f5d7\" (UID: \"31cac73a-d9f6-4a14-b4fc-d090af94f5d7\") "
May 26 11:54:21.185169 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.185143    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-success-200-isvc-5b0de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-5b0de-kube-rbac-proxy-sar-config") pod "31cac73a-d9f6-4a14-b4fc-d090af94f5d7" (UID: "31cac73a-d9f6-4a14-b4fc-d090af94f5d7"). InnerVolumeSpecName "success-200-isvc-5b0de-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:54:21.186722 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.186696    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31cac73a-d9f6-4a14-b4fc-d090af94f5d7" (UID: "31cac73a-d9f6-4a14-b4fc-d090af94f5d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:54:21.186835 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.186804    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-kube-api-access-kd9c2" (OuterVolumeSpecName: "kube-api-access-kd9c2") pod "31cac73a-d9f6-4a14-b4fc-d090af94f5d7" (UID: "31cac73a-d9f6-4a14-b4fc-d090af94f5d7"). InnerVolumeSpecName "kube-api-access-kd9c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:54:21.286175 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.286139    2563 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-5b0de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-success-200-isvc-5b0de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:54:21.286175 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.286170    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd9c2\" (UniqueName: \"kubernetes.io/projected/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-kube-api-access-kd9c2\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:54:21.286175 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.286181    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31cac73a-d9f6-4a14-b4fc-d090af94f5d7-proxy-tls\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:54:21.610103 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.610065    2563 generic.go:358] "Generic (PLEG): container finished" podID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" containerID="bf845ebfdb52f31bca755abe985ac257af308297653deced6d5a8e89a0ce8688" exitCode=0
May 26 11:54:21.610531 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.610136    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"
May 26 11:54:21.610531 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.610145    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng" event={"ID":"31cac73a-d9f6-4a14-b4fc-d090af94f5d7","Type":"ContainerDied","Data":"bf845ebfdb52f31bca755abe985ac257af308297653deced6d5a8e89a0ce8688"}
May 26 11:54:21.610531 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.610186    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng" event={"ID":"31cac73a-d9f6-4a14-b4fc-d090af94f5d7","Type":"ContainerDied","Data":"e07ed1390605925fea8e6b446d66ecf628ef5b358c4dc6b570be7a986b354934"}
May 26 11:54:21.610531 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.610203    2563 scope.go:117] "RemoveContainer" containerID="a14861bdef22b6e403f78930fe06a36e97097c544f656ed5b16e2691e5094f33"
May 26 11:54:21.610750 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.610679    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x" podUID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 26 11:54:21.618768 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.618749    2563 scope.go:117] "RemoveContainer" containerID="bf845ebfdb52f31bca755abe985ac257af308297653deced6d5a8e89a0ce8688"
May 26 11:54:21.625850 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.625834    2563 scope.go:117] "RemoveContainer" containerID="a14861bdef22b6e403f78930fe06a36e97097c544f656ed5b16e2691e5094f33"
May 26 11:54:21.626103 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:54:21.626086    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14861bdef22b6e403f78930fe06a36e97097c544f656ed5b16e2691e5094f33\": container with ID starting with a14861bdef22b6e403f78930fe06a36e97097c544f656ed5b16e2691e5094f33 not found: ID does not exist" containerID="a14861bdef22b6e403f78930fe06a36e97097c544f656ed5b16e2691e5094f33"
May 26 11:54:21.626145 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.626115    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14861bdef22b6e403f78930fe06a36e97097c544f656ed5b16e2691e5094f33"} err="failed to get container status \"a14861bdef22b6e403f78930fe06a36e97097c544f656ed5b16e2691e5094f33\": rpc error: code = NotFound desc = could not find container \"a14861bdef22b6e403f78930fe06a36e97097c544f656ed5b16e2691e5094f33\": container with ID starting with a14861bdef22b6e403f78930fe06a36e97097c544f656ed5b16e2691e5094f33 not found: ID does not exist"
May 26 11:54:21.626145 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.626132    2563 scope.go:117] "RemoveContainer" containerID="bf845ebfdb52f31bca755abe985ac257af308297653deced6d5a8e89a0ce8688"
May 26 11:54:21.626352 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:54:21.626329    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf845ebfdb52f31bca755abe985ac257af308297653deced6d5a8e89a0ce8688\": container with ID starting with bf845ebfdb52f31bca755abe985ac257af308297653deced6d5a8e89a0ce8688 not found: ID does not exist" containerID="bf845ebfdb52f31bca755abe985ac257af308297653deced6d5a8e89a0ce8688"
May 26 11:54:21.626451 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.626354    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf845ebfdb52f31bca755abe985ac257af308297653deced6d5a8e89a0ce8688"} err="failed to get container status \"bf845ebfdb52f31bca755abe985ac257af308297653deced6d5a8e89a0ce8688\": rpc error: code = NotFound desc = could not find container \"bf845ebfdb52f31bca755abe985ac257af308297653deced6d5a8e89a0ce8688\": container with ID starting with bf845ebfdb52f31bca755abe985ac257af308297653deced6d5a8e89a0ce8688 not found: ID does not exist"
May 26 11:54:21.632109 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.632083    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"]
May 26 11:54:21.639322 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.639296    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5b0de-predictor-7945cf7494-zdzng"]
May 26 11:54:21.796642 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:21.796604    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" path="/var/lib/kubelet/pods/31cac73a-d9f6-4a14-b4fc-d090af94f5d7/volumes"
May 26 11:54:26.505836 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:26.505798    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw" podUID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 26 11:54:26.615410 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:26.615381    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:54:26.615861 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:26.615834    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x" podUID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 26 11:54:36.505904 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:36.505870    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:54:36.616109 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:36.616069    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x" podUID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 26 11:54:46.616698 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:46.616658    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x" podUID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 26 11:54:56.616545 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:56.616505    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x" podUID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 26 11:54:58.366641 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.365398    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"]
May 26 11:54:58.366641 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.365764    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw" podUID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerName="kserve-container" containerID="cri-o://fa85650fe6b62aeac70fbe87fba116dace0badedc7998e525e1c8893d9ad80fd" gracePeriod=30
May 26 11:54:58.366641 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.365834    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw" podUID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerName="kube-rbac-proxy" containerID="cri-o://61cd67aaeb3e735e93871b6126dc84dafbb9ba517671d9bb71f6e7e24afdbf39" gracePeriod=30
May 26 11:54:58.390667 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.390639    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"]
May 26 11:54:58.391091 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.391071    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" containerName="kserve-container"
May 26 11:54:58.391091 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.391092    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" containerName="kserve-container"
May 26 11:54:58.391233 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.391110    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" containerName="kube-rbac-proxy"
May 26 11:54:58.391233 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.391116    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" containerName="kube-rbac-proxy"
May 26 11:54:58.391233 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.391196    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" containerName="kube-rbac-proxy"
May 26 11:54:58.391233 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.391211    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="31cac73a-d9f6-4a14-b4fc-d090af94f5d7" containerName="kserve-container"
May 26 11:54:58.394114 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.394091    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 11:54:58.396478 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.396457    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0a5df-kube-rbac-proxy-sar-config\""
May 26 11:54:58.396620 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.396598    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0a5df-predictor-serving-cert\""
May 26 11:54:58.405144 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.405126    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"]
May 26 11:54:58.509270 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.509234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c39fc2-6c59-46a7-b92c-0fa902ede916-proxy-tls\") pod \"success-200-isvc-0a5df-predictor-54955c646-lbvtd\" (UID: \"02c39fc2-6c59-46a7-b92c-0fa902ede916\") " pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 11:54:58.509463 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.509294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-0a5df-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02c39fc2-6c59-46a7-b92c-0fa902ede916-success-200-isvc-0a5df-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0a5df-predictor-54955c646-lbvtd\" (UID: \"02c39fc2-6c59-46a7-b92c-0fa902ede916\") " pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 11:54:58.509463 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.509446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmm6\" (UniqueName: \"kubernetes.io/projected/02c39fc2-6c59-46a7-b92c-0fa902ede916-kube-api-access-xnmm6\") pod \"success-200-isvc-0a5df-predictor-54955c646-lbvtd\" (UID: \"02c39fc2-6c59-46a7-b92c-0fa902ede916\") " pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 11:54:58.610615 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.610578    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmm6\" (UniqueName: \"kubernetes.io/projected/02c39fc2-6c59-46a7-b92c-0fa902ede916-kube-api-access-xnmm6\") pod \"success-200-isvc-0a5df-predictor-54955c646-lbvtd\" (UID: \"02c39fc2-6c59-46a7-b92c-0fa902ede916\") " pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 11:54:58.610793 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.610660    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c39fc2-6c59-46a7-b92c-0fa902ede916-proxy-tls\") pod \"success-200-isvc-0a5df-predictor-54955c646-lbvtd\" (UID: \"02c39fc2-6c59-46a7-b92c-0fa902ede916\") " pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 11:54:58.610793 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.610679    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-0a5df-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02c39fc2-6c59-46a7-b92c-0fa902ede916-success-200-isvc-0a5df-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0a5df-predictor-54955c646-lbvtd\" (UID: \"02c39fc2-6c59-46a7-b92c-0fa902ede916\") " pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 11:54:58.610918 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:54:58.610811    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-serving-cert: secret "success-200-isvc-0a5df-predictor-serving-cert" not found
May 26 11:54:58.610918 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:54:58.610885    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02c39fc2-6c59-46a7-b92c-0fa902ede916-proxy-tls podName:02c39fc2-6c59-46a7-b92c-0fa902ede916 nodeName:}" failed. No retries permitted until 2026-05-26 11:54:59.110864468 +0000 UTC m=+1831.916702921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/02c39fc2-6c59-46a7-b92c-0fa902ede916-proxy-tls") pod "success-200-isvc-0a5df-predictor-54955c646-lbvtd" (UID: "02c39fc2-6c59-46a7-b92c-0fa902ede916") : secret "success-200-isvc-0a5df-predictor-serving-cert" not found
May 26 11:54:58.611309 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.611264    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-0a5df-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02c39fc2-6c59-46a7-b92c-0fa902ede916-success-200-isvc-0a5df-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0a5df-predictor-54955c646-lbvtd\" (UID: \"02c39fc2-6c59-46a7-b92c-0fa902ede916\") " pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 11:54:58.621928 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.621863    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmm6\" (UniqueName: \"kubernetes.io/projected/02c39fc2-6c59-46a7-b92c-0fa902ede916-kube-api-access-xnmm6\") pod \"success-200-isvc-0a5df-predictor-54955c646-lbvtd\" (UID: \"02c39fc2-6c59-46a7-b92c-0fa902ede916\") " pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 11:54:58.743466 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.743427    2563 generic.go:358] "Generic (PLEG): container finished" podID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerID="61cd67aaeb3e735e93871b6126dc84dafbb9ba517671d9bb71f6e7e24afdbf39" exitCode=2
May 26 11:54:58.743647 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:58.743503    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw" event={"ID":"56b04d4a-e2b3-43aa-a6f1-56035cdb156d","Type":"ContainerDied","Data":"61cd67aaeb3e735e93871b6126dc84dafbb9ba517671d9bb71f6e7e24afdbf39"}
May 26 11:54:59.115749 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:59.115713    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c39fc2-6c59-46a7-b92c-0fa902ede916-proxy-tls\") pod \"success-200-isvc-0a5df-predictor-54955c646-lbvtd\" (UID: \"02c39fc2-6c59-46a7-b92c-0fa902ede916\") " pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 11:54:59.118160 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:59.118136    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c39fc2-6c59-46a7-b92c-0fa902ede916-proxy-tls\") pod \"success-200-isvc-0a5df-predictor-54955c646-lbvtd\" (UID: \"02c39fc2-6c59-46a7-b92c-0fa902ede916\") " pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 11:54:59.305953 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:59.305916    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 11:54:59.437128 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:59.437095    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"]
May 26 11:54:59.440331 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:54:59.440302    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c39fc2_6c59_46a7_b92c_0fa902ede916.slice/crio-719c0f8872bd86fdcd9e295753f6d9b47666f8b44d9b0e8a99589dcb6c8c89a2 WatchSource:0}: Error finding container 719c0f8872bd86fdcd9e295753f6d9b47666f8b44d9b0e8a99589dcb6c8c89a2: Status 404 returned error can't find the container with id 719c0f8872bd86fdcd9e295753f6d9b47666f8b44d9b0e8a99589dcb6c8c89a2
May 26 11:54:59.749276 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:59.749239    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd" event={"ID":"02c39fc2-6c59-46a7-b92c-0fa902ede916","Type":"ContainerStarted","Data":"20e5dfee5ccc3cf1f37a0528d0730d9a32afa00d2bb19e39a9f69a1082665f3a"}
May 26 11:54:59.749466 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:59.749306    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd" event={"ID":"02c39fc2-6c59-46a7-b92c-0fa902ede916","Type":"ContainerStarted","Data":"09d909396536f6596cfc40394885b4a4f8cb6464266e2ac2903a189f18bcf8d5"}
May 26 11:54:59.749466 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:59.749320    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd" event={"ID":"02c39fc2-6c59-46a7-b92c-0fa902ede916","Type":"ContainerStarted","Data":"719c0f8872bd86fdcd9e295753f6d9b47666f8b44d9b0e8a99589dcb6c8c89a2"}
May 26 11:54:59.749466 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:59.749403    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 11:54:59.769826 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:54:59.769788    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd" podStartSLOduration=1.7697753779999998 podStartE2EDuration="1.769775378s" podCreationTimestamp="2026-05-26 11:54:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:54:59.768190794 +0000 UTC m=+1832.574029260" watchObservedRunningTime="2026-05-26 11:54:59.769775378 +0000 UTC m=+1832.575613842"
May 26 11:55:00.753594 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:00.753555    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 11:55:00.754854 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:00.754822    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd" podUID="02c39fc2-6c59-46a7-b92c-0fa902ede916" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 26 11:55:01.501365 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.501313    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw" podUID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
May 26 11:55:01.710104 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.710080    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:55:01.757803 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.757711    2563 generic.go:358] "Generic (PLEG): container finished" podID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerID="fa85650fe6b62aeac70fbe87fba116dace0badedc7998e525e1c8893d9ad80fd" exitCode=0
May 26 11:55:01.757803 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.757787    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw" event={"ID":"56b04d4a-e2b3-43aa-a6f1-56035cdb156d","Type":"ContainerDied","Data":"fa85650fe6b62aeac70fbe87fba116dace0badedc7998e525e1c8893d9ad80fd"}
May 26 11:55:01.758269 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.757800    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"
May 26 11:55:01.758269 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.757823    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw" event={"ID":"56b04d4a-e2b3-43aa-a6f1-56035cdb156d","Type":"ContainerDied","Data":"a48cb8aba3ca424dbd047de944dc41936040977fff40b69d45f9dd23fa7c7e0b"}
May 26 11:55:01.758269 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.757843    2563 scope.go:117] "RemoveContainer" containerID="61cd67aaeb3e735e93871b6126dc84dafbb9ba517671d9bb71f6e7e24afdbf39"
May 26 11:55:01.758437 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.758258    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd" podUID="02c39fc2-6c59-46a7-b92c-0fa902ede916" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 26 11:55:01.766164 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.766142    2563 scope.go:117] "RemoveContainer" containerID="fa85650fe6b62aeac70fbe87fba116dace0badedc7998e525e1c8893d9ad80fd"
May 26 11:55:01.773860 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.773842    2563 scope.go:117] "RemoveContainer" containerID="61cd67aaeb3e735e93871b6126dc84dafbb9ba517671d9bb71f6e7e24afdbf39"
May 26 11:55:01.774149 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:55:01.774114    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61cd67aaeb3e735e93871b6126dc84dafbb9ba517671d9bb71f6e7e24afdbf39\": container with ID starting with 61cd67aaeb3e735e93871b6126dc84dafbb9ba517671d9bb71f6e7e24afdbf39 not found: ID does not exist" containerID="61cd67aaeb3e735e93871b6126dc84dafbb9ba517671d9bb71f6e7e24afdbf39"
May 26 11:55:01.774254 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.774152    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61cd67aaeb3e735e93871b6126dc84dafbb9ba517671d9bb71f6e7e24afdbf39"} err="failed to get container status \"61cd67aaeb3e735e93871b6126dc84dafbb9ba517671d9bb71f6e7e24afdbf39\": rpc error: code = NotFound desc = could not find container \"61cd67aaeb3e735e93871b6126dc84dafbb9ba517671d9bb71f6e7e24afdbf39\": container with ID starting with 61cd67aaeb3e735e93871b6126dc84dafbb9ba517671d9bb71f6e7e24afdbf39 not found: ID does not exist"
May 26 11:55:01.774254 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.774175    2563 scope.go:117] "RemoveContainer" containerID="fa85650fe6b62aeac70fbe87fba116dace0badedc7998e525e1c8893d9ad80fd"
May 26 11:55:01.774439 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:55:01.774412    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa85650fe6b62aeac70fbe87fba116dace0badedc7998e525e1c8893d9ad80fd\": container with ID starting with fa85650fe6b62aeac70fbe87fba116dace0badedc7998e525e1c8893d9ad80fd not found: ID does not exist" containerID="fa85650fe6b62aeac70fbe87fba116dace0badedc7998e525e1c8893d9ad80fd"
May 26 11:55:01.774485 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.774449    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa85650fe6b62aeac70fbe87fba116dace0badedc7998e525e1c8893d9ad80fd"} err="failed to get container status \"fa85650fe6b62aeac70fbe87fba116dace0badedc7998e525e1c8893d9ad80fd\": rpc error: code = NotFound desc = could not find container \"fa85650fe6b62aeac70fbe87fba116dace0badedc7998e525e1c8893d9ad80fd\": container with ID starting with fa85650fe6b62aeac70fbe87fba116dace0badedc7998e525e1c8893d9ad80fd not found: ID does not exist"
May 26 11:55:01.841479 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.841452    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58wvw\" (UniqueName: \"kubernetes.io/projected/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-kube-api-access-58wvw\") pod \"56b04d4a-e2b3-43aa-a6f1-56035cdb156d\" (UID: \"56b04d4a-e2b3-43aa-a6f1-56035cdb156d\") "
May 26 11:55:01.841612 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.841551    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-proxy-tls\") pod \"56b04d4a-e2b3-43aa-a6f1-56035cdb156d\" (UID: \"56b04d4a-e2b3-43aa-a6f1-56035cdb156d\") "
May 26 11:55:01.841612 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.841580    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-5380a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-success-200-isvc-5380a-kube-rbac-proxy-sar-config\") pod \"56b04d4a-e2b3-43aa-a6f1-56035cdb156d\" (UID: \"56b04d4a-e2b3-43aa-a6f1-56035cdb156d\") "
May 26 11:55:01.841976 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.841946    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-success-200-isvc-5380a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-5380a-kube-rbac-proxy-sar-config") pod "56b04d4a-e2b3-43aa-a6f1-56035cdb156d" (UID: "56b04d4a-e2b3-43aa-a6f1-56035cdb156d"). InnerVolumeSpecName "success-200-isvc-5380a-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:55:01.843456 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.843432    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-kube-api-access-58wvw" (OuterVolumeSpecName: "kube-api-access-58wvw") pod "56b04d4a-e2b3-43aa-a6f1-56035cdb156d" (UID: "56b04d4a-e2b3-43aa-a6f1-56035cdb156d"). InnerVolumeSpecName "kube-api-access-58wvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:55:01.843725 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.843515    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "56b04d4a-e2b3-43aa-a6f1-56035cdb156d" (UID: "56b04d4a-e2b3-43aa-a6f1-56035cdb156d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:55:01.942245 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.942208    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58wvw\" (UniqueName: \"kubernetes.io/projected/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-kube-api-access-58wvw\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:55:01.942245 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.942245    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-proxy-tls\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:55:01.942522 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:01.942263    2563 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-5380a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56b04d4a-e2b3-43aa-a6f1-56035cdb156d-success-200-isvc-5380a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:55:02.081404 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:02.081372    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"]
May 26 11:55:02.085555 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:02.085523    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5380a-predictor-9b6ccbb7b-xkmbw"]
May 26 11:55:03.796685 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:03.796651    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" path="/var/lib/kubelet/pods/56b04d4a-e2b3-43aa-a6f1-56035cdb156d/volumes"
May 26 11:55:06.617048 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:06.617017    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:55:06.762833 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:06.762801    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 11:55:06.763381 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:06.763351    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd" podUID="02c39fc2-6c59-46a7-b92c-0fa902ede916" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 26 11:55:16.763365 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:16.763324    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd" podUID="02c39fc2-6c59-46a7-b92c-0fa902ede916" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 26 11:55:26.763580 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:26.763540    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd" podUID="02c39fc2-6c59-46a7-b92c-0fa902ede916" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 26 11:55:28.170913 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.170829    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"]
May 26 11:55:28.171444 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.171181    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x" podUID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" containerName="kserve-container" containerID="cri-o://e7e861287d27562041147fe752035c5da2acc7f39b0cc4c6a4f62d9b411cbff4" gracePeriod=30
May 26 11:55:28.171647 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.171593    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x" podUID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" containerName="kube-rbac-proxy" containerID="cri-o://28ea78423059c350e5c294275f521f22b1d4e252e6d00f4f826d73b39ea7b324" gracePeriod=30
May 26 11:55:28.196349 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.196318    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"]
May 26 11:55:28.196932 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.196917    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerName="kube-rbac-proxy"
May 26 11:55:28.196978 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.196936    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerName="kube-rbac-proxy"
May 26 11:55:28.196978 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.196970    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerName="kserve-container"
May 26 11:55:28.197040 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.196979    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerName="kserve-container"
May 26 11:55:28.197080 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.197069    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerName="kube-rbac-proxy"
May 26 11:55:28.197119 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.197084    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="56b04d4a-e2b3-43aa-a6f1-56035cdb156d" containerName="kserve-container"
May 26 11:55:28.202076 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.202056    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 11:55:28.204485 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.204466    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3c458-kube-rbac-proxy-sar-config\""
May 26 11:55:28.204648 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.204480    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3c458-predictor-serving-cert\""
May 26 11:55:28.210727 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.209687    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"]
May 26 11:55:28.256271 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.256240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf5vd\" (UniqueName: \"kubernetes.io/projected/65e63c61-bd7f-4e6b-b2f8-779a68944c37-kube-api-access-gf5vd\") pod \"success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v\" (UID: \"65e63c61-bd7f-4e6b-b2f8-779a68944c37\") " pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 11:55:28.256475 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.256298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e63c61-bd7f-4e6b-b2f8-779a68944c37-proxy-tls\") pod \"success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v\" (UID: \"65e63c61-bd7f-4e6b-b2f8-779a68944c37\") " pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 11:55:28.256475 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.256363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-3c458-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65e63c61-bd7f-4e6b-b2f8-779a68944c37-success-200-isvc-3c458-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v\" (UID: \"65e63c61-bd7f-4e6b-b2f8-779a68944c37\") " pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 11:55:28.357128 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.357091    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gf5vd\" (UniqueName: \"kubernetes.io/projected/65e63c61-bd7f-4e6b-b2f8-779a68944c37-kube-api-access-gf5vd\") pod \"success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v\" (UID: \"65e63c61-bd7f-4e6b-b2f8-779a68944c37\") " pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 11:55:28.357128 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.357132    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e63c61-bd7f-4e6b-b2f8-779a68944c37-proxy-tls\") pod \"success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v\" (UID: \"65e63c61-bd7f-4e6b-b2f8-779a68944c37\") " pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 11:55:28.357395 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:55:28.357229    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-3c458-predictor-serving-cert: secret "success-200-isvc-3c458-predictor-serving-cert" not found
May 26 11:55:28.357395 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.357277    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-3c458-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65e63c61-bd7f-4e6b-b2f8-779a68944c37-success-200-isvc-3c458-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v\" (UID: \"65e63c61-bd7f-4e6b-b2f8-779a68944c37\") " pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 11:55:28.357395 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:55:28.357327    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65e63c61-bd7f-4e6b-b2f8-779a68944c37-proxy-tls podName:65e63c61-bd7f-4e6b-b2f8-779a68944c37 nodeName:}" failed. No retries permitted until 2026-05-26 11:55:28.857305097 +0000 UTC m=+1861.663143553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/65e63c61-bd7f-4e6b-b2f8-779a68944c37-proxy-tls") pod "success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v" (UID: "65e63c61-bd7f-4e6b-b2f8-779a68944c37") : secret "success-200-isvc-3c458-predictor-serving-cert" not found
May 26 11:55:28.358015 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.357991    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-3c458-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65e63c61-bd7f-4e6b-b2f8-779a68944c37-success-200-isvc-3c458-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v\" (UID: \"65e63c61-bd7f-4e6b-b2f8-779a68944c37\") " pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 11:55:28.365769 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.365742    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf5vd\" (UniqueName: \"kubernetes.io/projected/65e63c61-bd7f-4e6b-b2f8-779a68944c37-kube-api-access-gf5vd\") pod \"success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v\" (UID: \"65e63c61-bd7f-4e6b-b2f8-779a68944c37\") " pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 11:55:28.849201 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.849164    2563 generic.go:358] "Generic (PLEG): container finished" podID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" containerID="28ea78423059c350e5c294275f521f22b1d4e252e6d00f4f826d73b39ea7b324" exitCode=2
May 26 11:55:28.849378 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.849242    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x" event={"ID":"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b","Type":"ContainerDied","Data":"28ea78423059c350e5c294275f521f22b1d4e252e6d00f4f826d73b39ea7b324"}
May 26 11:55:28.861631 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.861606    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e63c61-bd7f-4e6b-b2f8-779a68944c37-proxy-tls\") pod \"success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v\" (UID: \"65e63c61-bd7f-4e6b-b2f8-779a68944c37\") " pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 11:55:28.864016 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:28.863997    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e63c61-bd7f-4e6b-b2f8-779a68944c37-proxy-tls\") pod \"success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v\" (UID: \"65e63c61-bd7f-4e6b-b2f8-779a68944c37\") " pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 11:55:29.114831 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:29.114742    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 11:55:29.250165 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:29.250127    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"]
May 26 11:55:29.253719 ip-10-0-135-71 kubenswrapper[2563]: W0526 11:55:29.253673    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e63c61_bd7f_4e6b_b2f8_779a68944c37.slice/crio-798b95d699c4a0a066d3e0831aeed8b94cd828b7b79ca63ca5a381a17a5c3f78 WatchSource:0}: Error finding container 798b95d699c4a0a066d3e0831aeed8b94cd828b7b79ca63ca5a381a17a5c3f78: Status 404 returned error can't find the container with id 798b95d699c4a0a066d3e0831aeed8b94cd828b7b79ca63ca5a381a17a5c3f78
May 26 11:55:29.854263 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:29.854223    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v" event={"ID":"65e63c61-bd7f-4e6b-b2f8-779a68944c37","Type":"ContainerStarted","Data":"bd362f09904ccbcabdf9cf2c34804f33242155a58cc0ff186582e29898d6ac64"}
May 26 11:55:29.854263 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:29.854266    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v" event={"ID":"65e63c61-bd7f-4e6b-b2f8-779a68944c37","Type":"ContainerStarted","Data":"f279a87b11993b1d8a080817a28d303d302b2d4045a26186b13fa80ddbc5877a"}
May 26 11:55:29.854471 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:29.854278    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v" event={"ID":"65e63c61-bd7f-4e6b-b2f8-779a68944c37","Type":"ContainerStarted","Data":"798b95d699c4a0a066d3e0831aeed8b94cd828b7b79ca63ca5a381a17a5c3f78"}
May 26 11:55:29.854471 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:29.854387    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 11:55:29.873449 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:29.873405    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v" podStartSLOduration=1.873391236 podStartE2EDuration="1.873391236s" podCreationTimestamp="2026-05-26 11:55:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:55:29.871961177 +0000 UTC m=+1862.677799668" watchObservedRunningTime="2026-05-26 11:55:29.873391236 +0000 UTC m=+1862.679229700"
May 26 11:55:30.858055 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:30.858014    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 11:55:30.859376 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:30.859341    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v" podUID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 26 11:55:31.315903 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.315881    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:55:31.384396 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.384324    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-fbdd2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-success-200-isvc-fbdd2-kube-rbac-proxy-sar-config\") pod \"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b\" (UID: \"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b\") "
May 26 11:55:31.384528 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.384407    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldktr\" (UniqueName: \"kubernetes.io/projected/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-kube-api-access-ldktr\") pod \"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b\" (UID: \"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b\") "
May 26 11:55:31.384528 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.384466    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-proxy-tls\") pod \"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b\" (UID: \"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b\") "
May 26 11:55:31.384750 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.384722    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-success-200-isvc-fbdd2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-fbdd2-kube-rbac-proxy-sar-config") pod "6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" (UID: "6399324a-aa3d-4bd2-ae41-36df7c5ccb5b"). InnerVolumeSpecName "success-200-isvc-fbdd2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:55:31.386456 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.386433    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" (UID: "6399324a-aa3d-4bd2-ae41-36df7c5ccb5b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:55:31.386574 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.386460    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-kube-api-access-ldktr" (OuterVolumeSpecName: "kube-api-access-ldktr") pod "6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" (UID: "6399324a-aa3d-4bd2-ae41-36df7c5ccb5b"). InnerVolumeSpecName "kube-api-access-ldktr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:55:31.485887 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.485856    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-proxy-tls\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:55:31.485887 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.485884    2563 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-fbdd2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-success-200-isvc-fbdd2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:55:31.485887 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.485894    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldktr\" (UniqueName: \"kubernetes.io/projected/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b-kube-api-access-ldktr\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 11:55:31.862945 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.862912    2563 generic.go:358] "Generic (PLEG): container finished" podID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" containerID="e7e861287d27562041147fe752035c5da2acc7f39b0cc4c6a4f62d9b411cbff4" exitCode=0
May 26 11:55:31.863410 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.863002    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"
May 26 11:55:31.863410 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.863000    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x" event={"ID":"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b","Type":"ContainerDied","Data":"e7e861287d27562041147fe752035c5da2acc7f39b0cc4c6a4f62d9b411cbff4"}
May 26 11:55:31.863410 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.863108    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x" event={"ID":"6399324a-aa3d-4bd2-ae41-36df7c5ccb5b","Type":"ContainerDied","Data":"e97eb01a78074a5612bc7f61378235094300293d80d9c54ec63397669db81442"}
May 26 11:55:31.863410 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.863128    2563 scope.go:117] "RemoveContainer" containerID="28ea78423059c350e5c294275f521f22b1d4e252e6d00f4f826d73b39ea7b324"
May 26 11:55:31.863681 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.863651    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v" podUID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 26 11:55:31.871317 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.871239    2563 scope.go:117] "RemoveContainer" containerID="e7e861287d27562041147fe752035c5da2acc7f39b0cc4c6a4f62d9b411cbff4"
May 26 11:55:31.877985 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.877966    2563 scope.go:117] "RemoveContainer" containerID="28ea78423059c350e5c294275f521f22b1d4e252e6d00f4f826d73b39ea7b324"
May 26 11:55:31.878222 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:55:31.878204    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ea78423059c350e5c294275f521f22b1d4e252e6d00f4f826d73b39ea7b324\": container with ID starting with 28ea78423059c350e5c294275f521f22b1d4e252e6d00f4f826d73b39ea7b324 not found: ID does not exist" containerID="28ea78423059c350e5c294275f521f22b1d4e252e6d00f4f826d73b39ea7b324"
May 26 11:55:31.878303 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.878233    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ea78423059c350e5c294275f521f22b1d4e252e6d00f4f826d73b39ea7b324"} err="failed to get container status \"28ea78423059c350e5c294275f521f22b1d4e252e6d00f4f826d73b39ea7b324\": rpc error: code = NotFound desc = could not find container \"28ea78423059c350e5c294275f521f22b1d4e252e6d00f4f826d73b39ea7b324\": container with ID starting with 28ea78423059c350e5c294275f521f22b1d4e252e6d00f4f826d73b39ea7b324 not found: ID does not exist"
May 26 11:55:31.878303 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.878257    2563 scope.go:117] "RemoveContainer" containerID="e7e861287d27562041147fe752035c5da2acc7f39b0cc4c6a4f62d9b411cbff4"
May 26 11:55:31.879584 ip-10-0-135-71 kubenswrapper[2563]: E0526 11:55:31.879527    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e861287d27562041147fe752035c5da2acc7f39b0cc4c6a4f62d9b411cbff4\": container with ID starting with e7e861287d27562041147fe752035c5da2acc7f39b0cc4c6a4f62d9b411cbff4 not found: ID does not exist" containerID="e7e861287d27562041147fe752035c5da2acc7f39b0cc4c6a4f62d9b411cbff4"
May 26 11:55:31.879584 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.879560    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e861287d27562041147fe752035c5da2acc7f39b0cc4c6a4f62d9b411cbff4"} err="failed to get container status \"e7e861287d27562041147fe752035c5da2acc7f39b0cc4c6a4f62d9b411cbff4\": rpc error: code = NotFound desc = could not find container \"e7e861287d27562041147fe752035c5da2acc7f39b0cc4c6a4f62d9b411cbff4\": container with ID starting with e7e861287d27562041147fe752035c5da2acc7f39b0cc4c6a4f62d9b411cbff4 not found: ID does not exist"
May 26 11:55:31.881082 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.881060    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"]
May 26 11:55:31.884629 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:31.884608    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbdd2-predictor-6f8b675bfd-s6q7x"]
May 26 11:55:33.796721 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:33.796675    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" path="/var/lib/kubelet/pods/6399324a-aa3d-4bd2-ae41-36df7c5ccb5b/volumes"
May 26 11:55:36.764319 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:36.764259    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd" podUID="02c39fc2-6c59-46a7-b92c-0fa902ede916" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 26 11:55:36.868512 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:36.868474    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 11:55:36.868981 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:36.868949    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v" podUID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 26 11:55:46.764879 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:46.764850    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 11:55:46.869878 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:46.869835    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v" podUID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 26 11:55:56.869690 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:55:56.869644    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v" podUID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 26 11:56:06.869393 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:56:06.869355    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v" podUID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 26 11:56:16.870299 ip-10-0-135-71 kubenswrapper[2563]: I0526 11:56:16.870259    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 12:04:13.312216 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.312176    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"]
May 26 12:04:13.314806 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.312596    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd" podUID="02c39fc2-6c59-46a7-b92c-0fa902ede916" containerName="kserve-container" containerID="cri-o://09d909396536f6596cfc40394885b4a4f8cb6464266e2ac2903a189f18bcf8d5" gracePeriod=30
May 26 12:04:13.314806 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.312631    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd" podUID="02c39fc2-6c59-46a7-b92c-0fa902ede916" containerName="kube-rbac-proxy" containerID="cri-o://20e5dfee5ccc3cf1f37a0528d0730d9a32afa00d2bb19e39a9f69a1082665f3a" gracePeriod=30
May 26 12:04:13.419937 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.419903    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"]
May 26 12:04:13.420404 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.420386    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" containerName="kserve-container"
May 26 12:04:13.420510 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.420407    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" containerName="kserve-container"
May 26 12:04:13.420510 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.420456    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" containerName="kube-rbac-proxy"
May 26 12:04:13.420510 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.420466    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" containerName="kube-rbac-proxy"
May 26 12:04:13.420676 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.420556    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" containerName="kube-rbac-proxy"
May 26 12:04:13.420676 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.420577    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="6399324a-aa3d-4bd2-ae41-36df7c5ccb5b" containerName="kserve-container"
May 26 12:04:13.424384 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.424359    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:04:13.428207 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.427633    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-81833-predictor-serving-cert\""
May 26 12:04:13.428207 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.427956    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-81833-kube-rbac-proxy-sar-config\""
May 26 12:04:13.442758 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.442733    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"]
May 26 12:04:13.502102 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.502071    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85hrr\" (UniqueName: \"kubernetes.io/projected/25b5fa81-3e00-4738-b260-086600720195-kube-api-access-85hrr\") pod \"success-200-isvc-81833-predictor-5554ddb76-qccpc\" (UID: \"25b5fa81-3e00-4738-b260-086600720195\") " pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:04:13.502102 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.502107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25b5fa81-3e00-4738-b260-086600720195-proxy-tls\") pod \"success-200-isvc-81833-predictor-5554ddb76-qccpc\" (UID: \"25b5fa81-3e00-4738-b260-086600720195\") " pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:04:13.502332 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.502261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-81833-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25b5fa81-3e00-4738-b260-086600720195-success-200-isvc-81833-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-81833-predictor-5554ddb76-qccpc\" (UID: \"25b5fa81-3e00-4738-b260-086600720195\") " pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:04:13.603519 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.603414    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-85hrr\" (UniqueName: \"kubernetes.io/projected/25b5fa81-3e00-4738-b260-086600720195-kube-api-access-85hrr\") pod \"success-200-isvc-81833-predictor-5554ddb76-qccpc\" (UID: \"25b5fa81-3e00-4738-b260-086600720195\") " pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:04:13.603519 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.603454    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25b5fa81-3e00-4738-b260-086600720195-proxy-tls\") pod \"success-200-isvc-81833-predictor-5554ddb76-qccpc\" (UID: \"25b5fa81-3e00-4738-b260-086600720195\") " pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:04:13.603763 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.603536    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-81833-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25b5fa81-3e00-4738-b260-086600720195-success-200-isvc-81833-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-81833-predictor-5554ddb76-qccpc\" (UID: \"25b5fa81-3e00-4738-b260-086600720195\") " pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:04:13.603823 ip-10-0-135-71 kubenswrapper[2563]: E0526 12:04:13.603783    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-81833-predictor-serving-cert: secret "success-200-isvc-81833-predictor-serving-cert" not found
May 26 12:04:13.603887 ip-10-0-135-71 kubenswrapper[2563]: E0526 12:04:13.603860    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b5fa81-3e00-4738-b260-086600720195-proxy-tls podName:25b5fa81-3e00-4738-b260-086600720195 nodeName:}" failed. No retries permitted until 2026-05-26 12:04:14.103839044 +0000 UTC m=+2386.909677502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/25b5fa81-3e00-4738-b260-086600720195-proxy-tls") pod "success-200-isvc-81833-predictor-5554ddb76-qccpc" (UID: "25b5fa81-3e00-4738-b260-086600720195") : secret "success-200-isvc-81833-predictor-serving-cert" not found
May 26 12:04:13.604248 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.604228    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-81833-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25b5fa81-3e00-4738-b260-086600720195-success-200-isvc-81833-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-81833-predictor-5554ddb76-qccpc\" (UID: \"25b5fa81-3e00-4738-b260-086600720195\") " pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:04:13.616704 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.616675    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85hrr\" (UniqueName: \"kubernetes.io/projected/25b5fa81-3e00-4738-b260-086600720195-kube-api-access-85hrr\") pod \"success-200-isvc-81833-predictor-5554ddb76-qccpc\" (UID: \"25b5fa81-3e00-4738-b260-086600720195\") " pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:04:13.701145 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.701112    2563 generic.go:358] "Generic (PLEG): container finished" podID="02c39fc2-6c59-46a7-b92c-0fa902ede916" containerID="20e5dfee5ccc3cf1f37a0528d0730d9a32afa00d2bb19e39a9f69a1082665f3a" exitCode=2
May 26 12:04:13.701369 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:13.701183    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd" event={"ID":"02c39fc2-6c59-46a7-b92c-0fa902ede916","Type":"ContainerDied","Data":"20e5dfee5ccc3cf1f37a0528d0730d9a32afa00d2bb19e39a9f69a1082665f3a"}
May 26 12:04:14.108046 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:14.108001    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25b5fa81-3e00-4738-b260-086600720195-proxy-tls\") pod \"success-200-isvc-81833-predictor-5554ddb76-qccpc\" (UID: \"25b5fa81-3e00-4738-b260-086600720195\") " pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:04:14.110474 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:14.110443    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25b5fa81-3e00-4738-b260-086600720195-proxy-tls\") pod \"success-200-isvc-81833-predictor-5554ddb76-qccpc\" (UID: \"25b5fa81-3e00-4738-b260-086600720195\") " pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:04:14.340255 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:14.340219    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:04:14.473894 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:14.473864    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"]
May 26 12:04:14.476589 ip-10-0-135-71 kubenswrapper[2563]: W0526 12:04:14.476562    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b5fa81_3e00_4738_b260_086600720195.slice/crio-68c40ab5f805049dbf420711386376c45dc73f56ec03ed01070f5d5f474fd093 WatchSource:0}: Error finding container 68c40ab5f805049dbf420711386376c45dc73f56ec03ed01070f5d5f474fd093: Status 404 returned error can't find the container with id 68c40ab5f805049dbf420711386376c45dc73f56ec03ed01070f5d5f474fd093
May 26 12:04:14.478391 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:14.478375    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 12:04:14.706625 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:14.706591    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc" event={"ID":"25b5fa81-3e00-4738-b260-086600720195","Type":"ContainerStarted","Data":"228e78bbf649b19afb2669a2c375df1fc9162c430ce23ca974578fb074061e8f"}
May 26 12:04:14.706786 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:14.706634    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc" event={"ID":"25b5fa81-3e00-4738-b260-086600720195","Type":"ContainerStarted","Data":"a1f393eebf4b21540ed949cc82d83eab589ac772ccca63c0a9ddc473f4f14d53"}
May 26 12:04:14.706786 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:14.706651    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc" event={"ID":"25b5fa81-3e00-4738-b260-086600720195","Type":"ContainerStarted","Data":"68c40ab5f805049dbf420711386376c45dc73f56ec03ed01070f5d5f474fd093"}
May 26 12:04:14.706786 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:14.706681    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:04:14.726205 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:14.726159    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc" podStartSLOduration=1.726146699 podStartE2EDuration="1.726146699s" podCreationTimestamp="2026-05-26 12:04:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:04:14.724261791 +0000 UTC m=+2387.530100249" watchObservedRunningTime="2026-05-26 12:04:14.726146699 +0000 UTC m=+2387.531985162"
May 26 12:04:15.710059 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:15.710020    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:04:15.711457 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:15.711427    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc" podUID="25b5fa81-3e00-4738-b260-086600720195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 26 12:04:16.562657 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.562631    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 12:04:16.628376 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.628299    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c39fc2-6c59-46a7-b92c-0fa902ede916-proxy-tls\") pod \"02c39fc2-6c59-46a7-b92c-0fa902ede916\" (UID: \"02c39fc2-6c59-46a7-b92c-0fa902ede916\") "
May 26 12:04:16.628376 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.628342    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-0a5df-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02c39fc2-6c59-46a7-b92c-0fa902ede916-success-200-isvc-0a5df-kube-rbac-proxy-sar-config\") pod \"02c39fc2-6c59-46a7-b92c-0fa902ede916\" (UID: \"02c39fc2-6c59-46a7-b92c-0fa902ede916\") "
May 26 12:04:16.628592 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.628386    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnmm6\" (UniqueName: \"kubernetes.io/projected/02c39fc2-6c59-46a7-b92c-0fa902ede916-kube-api-access-xnmm6\") pod \"02c39fc2-6c59-46a7-b92c-0fa902ede916\" (UID: \"02c39fc2-6c59-46a7-b92c-0fa902ede916\") "
May 26 12:04:16.628704 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.628677    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c39fc2-6c59-46a7-b92c-0fa902ede916-success-200-isvc-0a5df-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-0a5df-kube-rbac-proxy-sar-config") pod "02c39fc2-6c59-46a7-b92c-0fa902ede916" (UID: "02c39fc2-6c59-46a7-b92c-0fa902ede916"). InnerVolumeSpecName "success-200-isvc-0a5df-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:04:16.630320 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.630298    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c39fc2-6c59-46a7-b92c-0fa902ede916-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "02c39fc2-6c59-46a7-b92c-0fa902ede916" (UID: "02c39fc2-6c59-46a7-b92c-0fa902ede916"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:04:16.630457 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.630433    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c39fc2-6c59-46a7-b92c-0fa902ede916-kube-api-access-xnmm6" (OuterVolumeSpecName: "kube-api-access-xnmm6") pod "02c39fc2-6c59-46a7-b92c-0fa902ede916" (UID: "02c39fc2-6c59-46a7-b92c-0fa902ede916"). InnerVolumeSpecName "kube-api-access-xnmm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:04:16.715471 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.715433    2563 generic.go:358] "Generic (PLEG): container finished" podID="02c39fc2-6c59-46a7-b92c-0fa902ede916" containerID="09d909396536f6596cfc40394885b4a4f8cb6464266e2ac2903a189f18bcf8d5" exitCode=0
May 26 12:04:16.715919 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.715490    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd" event={"ID":"02c39fc2-6c59-46a7-b92c-0fa902ede916","Type":"ContainerDied","Data":"09d909396536f6596cfc40394885b4a4f8cb6464266e2ac2903a189f18bcf8d5"}
May 26 12:04:16.715919 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.715540    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd" event={"ID":"02c39fc2-6c59-46a7-b92c-0fa902ede916","Type":"ContainerDied","Data":"719c0f8872bd86fdcd9e295753f6d9b47666f8b44d9b0e8a99589dcb6c8c89a2"}
May 26 12:04:16.715919 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.715562    2563 scope.go:117] "RemoveContainer" containerID="20e5dfee5ccc3cf1f37a0528d0730d9a32afa00d2bb19e39a9f69a1082665f3a"
May 26 12:04:16.715919 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.715502    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"
May 26 12:04:16.716318 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.716264    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc" podUID="25b5fa81-3e00-4738-b260-086600720195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 26 12:04:16.723906 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.723886    2563 scope.go:117] "RemoveContainer" containerID="09d909396536f6596cfc40394885b4a4f8cb6464266e2ac2903a189f18bcf8d5"
May 26 12:04:16.730250 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.729364    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnmm6\" (UniqueName: \"kubernetes.io/projected/02c39fc2-6c59-46a7-b92c-0fa902ede916-kube-api-access-xnmm6\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 12:04:16.730250 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.729398    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c39fc2-6c59-46a7-b92c-0fa902ede916-proxy-tls\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 12:04:16.730250 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.729414    2563 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-0a5df-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02c39fc2-6c59-46a7-b92c-0fa902ede916-success-200-isvc-0a5df-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 12:04:16.738646 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.738590    2563 scope.go:117] "RemoveContainer" containerID="20e5dfee5ccc3cf1f37a0528d0730d9a32afa00d2bb19e39a9f69a1082665f3a"
May 26 12:04:16.738990 ip-10-0-135-71 kubenswrapper[2563]: E0526 12:04:16.738967    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e5dfee5ccc3cf1f37a0528d0730d9a32afa00d2bb19e39a9f69a1082665f3a\": container with ID starting with 20e5dfee5ccc3cf1f37a0528d0730d9a32afa00d2bb19e39a9f69a1082665f3a not found: ID does not exist" containerID="20e5dfee5ccc3cf1f37a0528d0730d9a32afa00d2bb19e39a9f69a1082665f3a"
May 26 12:04:16.739066 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.738999    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e5dfee5ccc3cf1f37a0528d0730d9a32afa00d2bb19e39a9f69a1082665f3a"} err="failed to get container status \"20e5dfee5ccc3cf1f37a0528d0730d9a32afa00d2bb19e39a9f69a1082665f3a\": rpc error: code = NotFound desc = could not find container \"20e5dfee5ccc3cf1f37a0528d0730d9a32afa00d2bb19e39a9f69a1082665f3a\": container with ID starting with 20e5dfee5ccc3cf1f37a0528d0730d9a32afa00d2bb19e39a9f69a1082665f3a not found: ID does not exist"
May 26 12:04:16.739066 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.739018    2563 scope.go:117] "RemoveContainer" containerID="09d909396536f6596cfc40394885b4a4f8cb6464266e2ac2903a189f18bcf8d5"
May 26 12:04:16.739354 ip-10-0-135-71 kubenswrapper[2563]: E0526 12:04:16.739327    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d909396536f6596cfc40394885b4a4f8cb6464266e2ac2903a189f18bcf8d5\": container with ID starting with 09d909396536f6596cfc40394885b4a4f8cb6464266e2ac2903a189f18bcf8d5 not found: ID does not exist" containerID="09d909396536f6596cfc40394885b4a4f8cb6464266e2ac2903a189f18bcf8d5"
May 26 12:04:16.739418 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.739366    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d909396536f6596cfc40394885b4a4f8cb6464266e2ac2903a189f18bcf8d5"} err="failed to get container status \"09d909396536f6596cfc40394885b4a4f8cb6464266e2ac2903a189f18bcf8d5\": rpc error: code = NotFound desc = could not find container \"09d909396536f6596cfc40394885b4a4f8cb6464266e2ac2903a189f18bcf8d5\": container with ID starting with 09d909396536f6596cfc40394885b4a4f8cb6464266e2ac2903a189f18bcf8d5 not found: ID does not exist"
May 26 12:04:16.740302 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.740263    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"]
May 26 12:04:16.744174 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:16.744150    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a5df-predictor-54955c646-lbvtd"]
May 26 12:04:17.796763 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:17.796727    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02c39fc2-6c59-46a7-b92c-0fa902ede916" path="/var/lib/kubelet/pods/02c39fc2-6c59-46a7-b92c-0fa902ede916/volumes"
May 26 12:04:21.720926 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:21.720898    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:04:21.721468 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:21.721440    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc" podUID="25b5fa81-3e00-4738-b260-086600720195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 26 12:04:31.722170 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:31.722129    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc" podUID="25b5fa81-3e00-4738-b260-086600720195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 26 12:04:41.722438 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:41.722397    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc" podUID="25b5fa81-3e00-4738-b260-086600720195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 26 12:04:42.909665 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:42.909624    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"]
May 26 12:04:42.910235 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:42.910042    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v" podUID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" containerName="kserve-container" containerID="cri-o://f279a87b11993b1d8a080817a28d303d302b2d4045a26186b13fa80ddbc5877a" gracePeriod=30
May 26 12:04:42.910235 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:42.910072    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v" podUID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" containerName="kube-rbac-proxy" containerID="cri-o://bd362f09904ccbcabdf9cf2c34804f33242155a58cc0ff186582e29898d6ac64" gracePeriod=30
May 26 12:04:42.939354 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:42.939315    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"]
May 26 12:04:42.939965 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:42.939943    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c39fc2-6c59-46a7-b92c-0fa902ede916" containerName="kube-rbac-proxy"
May 26 12:04:42.940030 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:42.939970    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c39fc2-6c59-46a7-b92c-0fa902ede916" containerName="kube-rbac-proxy"
May 26 12:04:42.940030 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:42.939990    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c39fc2-6c59-46a7-b92c-0fa902ede916" containerName="kserve-container"
May 26 12:04:42.940030 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:42.939999    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c39fc2-6c59-46a7-b92c-0fa902ede916" containerName="kserve-container"
May 26 12:04:42.940139 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:42.940089    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="02c39fc2-6c59-46a7-b92c-0fa902ede916" containerName="kube-rbac-proxy"
May 26 12:04:42.940139 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:42.940108    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="02c39fc2-6c59-46a7-b92c-0fa902ede916" containerName="kserve-container"
May 26 12:04:42.944600 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:42.944578    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:04:42.947466 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:42.947446    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-152bc-predictor-serving-cert\""
May 26 12:04:42.947588 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:42.947484    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-152bc-kube-rbac-proxy-sar-config\""
May 26 12:04:42.954448 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:42.954409    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"]
May 26 12:04:43.056425 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:43.056385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-152bc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fc22dc1-5779-4102-9260-8107f828eddb-success-200-isvc-152bc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-152bc-predictor-7456d7b59b-hxqks\" (UID: \"6fc22dc1-5779-4102-9260-8107f828eddb\") " pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:04:43.056590 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:43.056444    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fc22dc1-5779-4102-9260-8107f828eddb-proxy-tls\") pod \"success-200-isvc-152bc-predictor-7456d7b59b-hxqks\" (UID: \"6fc22dc1-5779-4102-9260-8107f828eddb\") " pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:04:43.056590 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:43.056475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfr5x\" (UniqueName: \"kubernetes.io/projected/6fc22dc1-5779-4102-9260-8107f828eddb-kube-api-access-gfr5x\") pod \"success-200-isvc-152bc-predictor-7456d7b59b-hxqks\" (UID: \"6fc22dc1-5779-4102-9260-8107f828eddb\") " pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:04:43.157243 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:43.157200    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-152bc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fc22dc1-5779-4102-9260-8107f828eddb-success-200-isvc-152bc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-152bc-predictor-7456d7b59b-hxqks\" (UID: \"6fc22dc1-5779-4102-9260-8107f828eddb\") " pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:04:43.157445 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:43.157250    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fc22dc1-5779-4102-9260-8107f828eddb-proxy-tls\") pod \"success-200-isvc-152bc-predictor-7456d7b59b-hxqks\" (UID: \"6fc22dc1-5779-4102-9260-8107f828eddb\") " pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:04:43.157445 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:43.157276    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gfr5x\" (UniqueName: \"kubernetes.io/projected/6fc22dc1-5779-4102-9260-8107f828eddb-kube-api-access-gfr5x\") pod \"success-200-isvc-152bc-predictor-7456d7b59b-hxqks\" (UID: \"6fc22dc1-5779-4102-9260-8107f828eddb\") " pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:04:43.157566 ip-10-0-135-71 kubenswrapper[2563]: E0526 12:04:43.157438    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-152bc-predictor-serving-cert: secret "success-200-isvc-152bc-predictor-serving-cert" not found
May 26 12:04:43.157566 ip-10-0-135-71 kubenswrapper[2563]: E0526 12:04:43.157519    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fc22dc1-5779-4102-9260-8107f828eddb-proxy-tls podName:6fc22dc1-5779-4102-9260-8107f828eddb nodeName:}" failed. No retries permitted until 2026-05-26 12:04:43.657497419 +0000 UTC m=+2416.463335868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6fc22dc1-5779-4102-9260-8107f828eddb-proxy-tls") pod "success-200-isvc-152bc-predictor-7456d7b59b-hxqks" (UID: "6fc22dc1-5779-4102-9260-8107f828eddb") : secret "success-200-isvc-152bc-predictor-serving-cert" not found
May 26 12:04:43.157893 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:43.157871    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-152bc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fc22dc1-5779-4102-9260-8107f828eddb-success-200-isvc-152bc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-152bc-predictor-7456d7b59b-hxqks\" (UID: \"6fc22dc1-5779-4102-9260-8107f828eddb\") " pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:04:43.166328 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:43.166263    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfr5x\" (UniqueName: \"kubernetes.io/projected/6fc22dc1-5779-4102-9260-8107f828eddb-kube-api-access-gfr5x\") pod \"success-200-isvc-152bc-predictor-7456d7b59b-hxqks\" (UID: \"6fc22dc1-5779-4102-9260-8107f828eddb\") " pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:04:43.661105 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:43.661061    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fc22dc1-5779-4102-9260-8107f828eddb-proxy-tls\") pod \"success-200-isvc-152bc-predictor-7456d7b59b-hxqks\" (UID: \"6fc22dc1-5779-4102-9260-8107f828eddb\") " pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:04:43.663549 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:43.663527    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fc22dc1-5779-4102-9260-8107f828eddb-proxy-tls\") pod \"success-200-isvc-152bc-predictor-7456d7b59b-hxqks\" (UID: \"6fc22dc1-5779-4102-9260-8107f828eddb\") " pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:04:43.827702 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:43.822604    2563 generic.go:358] "Generic (PLEG): container finished" podID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" containerID="bd362f09904ccbcabdf9cf2c34804f33242155a58cc0ff186582e29898d6ac64" exitCode=2
May 26 12:04:43.827702 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:43.822675    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v" event={"ID":"65e63c61-bd7f-4e6b-b2f8-779a68944c37","Type":"ContainerDied","Data":"bd362f09904ccbcabdf9cf2c34804f33242155a58cc0ff186582e29898d6ac64"}
May 26 12:04:43.856848 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:43.856802    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:04:43.988741 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:43.988709    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"]
May 26 12:04:43.992249 ip-10-0-135-71 kubenswrapper[2563]: W0526 12:04:43.992221    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc22dc1_5779_4102_9260_8107f828eddb.slice/crio-35e479b18ee452b3fb2e19372babc8258b5aae06b83aa2b7602e33a420c9bb11 WatchSource:0}: Error finding container 35e479b18ee452b3fb2e19372babc8258b5aae06b83aa2b7602e33a420c9bb11: Status 404 returned error can't find the container with id 35e479b18ee452b3fb2e19372babc8258b5aae06b83aa2b7602e33a420c9bb11
May 26 12:04:44.830562 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:44.830518    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks" event={"ID":"6fc22dc1-5779-4102-9260-8107f828eddb","Type":"ContainerStarted","Data":"b164e838e909c834747e05319d58de1f85af5aca52de972c5dcd40fa072b6c90"}
May 26 12:04:44.830562 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:44.830565    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks" event={"ID":"6fc22dc1-5779-4102-9260-8107f828eddb","Type":"ContainerStarted","Data":"ddf94291a8d830e947ea31e52ba3dfa54497201f84cee4621073592a9bc52eb3"}
May 26 12:04:44.830835 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:44.830579    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks" event={"ID":"6fc22dc1-5779-4102-9260-8107f828eddb","Type":"ContainerStarted","Data":"35e479b18ee452b3fb2e19372babc8258b5aae06b83aa2b7602e33a420c9bb11"}
May 26 12:04:44.830835 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:44.830703    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:04:44.830835 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:44.830738    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:04:44.832255 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:44.832220    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks" podUID="6fc22dc1-5779-4102-9260-8107f828eddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 26 12:04:44.849250 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:44.849206    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks" podStartSLOduration=2.849192373 podStartE2EDuration="2.849192373s" podCreationTimestamp="2026-05-26 12:04:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:04:44.847751869 +0000 UTC m=+2417.653590335" watchObservedRunningTime="2026-05-26 12:04:44.849192373 +0000 UTC m=+2417.655030836"
May 26 12:04:45.835218 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:45.835174    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks" podUID="6fc22dc1-5779-4102-9260-8107f828eddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 26 12:04:46.159794 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.159771    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 12:04:46.183246 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.183215    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-3c458-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65e63c61-bd7f-4e6b-b2f8-779a68944c37-success-200-isvc-3c458-kube-rbac-proxy-sar-config\") pod \"65e63c61-bd7f-4e6b-b2f8-779a68944c37\" (UID: \"65e63c61-bd7f-4e6b-b2f8-779a68944c37\") "
May 26 12:04:46.183389 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.183326    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e63c61-bd7f-4e6b-b2f8-779a68944c37-proxy-tls\") pod \"65e63c61-bd7f-4e6b-b2f8-779a68944c37\" (UID: \"65e63c61-bd7f-4e6b-b2f8-779a68944c37\") "
May 26 12:04:46.183439 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.183396    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf5vd\" (UniqueName: \"kubernetes.io/projected/65e63c61-bd7f-4e6b-b2f8-779a68944c37-kube-api-access-gf5vd\") pod \"65e63c61-bd7f-4e6b-b2f8-779a68944c37\" (UID: \"65e63c61-bd7f-4e6b-b2f8-779a68944c37\") "
May 26 12:04:46.183597 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.183567    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e63c61-bd7f-4e6b-b2f8-779a68944c37-success-200-isvc-3c458-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-3c458-kube-rbac-proxy-sar-config") pod "65e63c61-bd7f-4e6b-b2f8-779a68944c37" (UID: "65e63c61-bd7f-4e6b-b2f8-779a68944c37"). InnerVolumeSpecName "success-200-isvc-3c458-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:04:46.185369 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.185346    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e63c61-bd7f-4e6b-b2f8-779a68944c37-kube-api-access-gf5vd" (OuterVolumeSpecName: "kube-api-access-gf5vd") pod "65e63c61-bd7f-4e6b-b2f8-779a68944c37" (UID: "65e63c61-bd7f-4e6b-b2f8-779a68944c37"). InnerVolumeSpecName "kube-api-access-gf5vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:04:46.185435 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.185425    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e63c61-bd7f-4e6b-b2f8-779a68944c37-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "65e63c61-bd7f-4e6b-b2f8-779a68944c37" (UID: "65e63c61-bd7f-4e6b-b2f8-779a68944c37"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:04:46.284367 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.284327    2563 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-3c458-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65e63c61-bd7f-4e6b-b2f8-779a68944c37-success-200-isvc-3c458-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 12:04:46.284367 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.284361    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e63c61-bd7f-4e6b-b2f8-779a68944c37-proxy-tls\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 12:04:46.284367 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.284374    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf5vd\" (UniqueName: \"kubernetes.io/projected/65e63c61-bd7f-4e6b-b2f8-779a68944c37-kube-api-access-gf5vd\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 12:04:46.840755 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.840719    2563 generic.go:358] "Generic (PLEG): container finished" podID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" containerID="f279a87b11993b1d8a080817a28d303d302b2d4045a26186b13fa80ddbc5877a" exitCode=0
May 26 12:04:46.841135 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.840786    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"
May 26 12:04:46.841135 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.840793    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v" event={"ID":"65e63c61-bd7f-4e6b-b2f8-779a68944c37","Type":"ContainerDied","Data":"f279a87b11993b1d8a080817a28d303d302b2d4045a26186b13fa80ddbc5877a"}
May 26 12:04:46.841135 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.840825    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v" event={"ID":"65e63c61-bd7f-4e6b-b2f8-779a68944c37","Type":"ContainerDied","Data":"798b95d699c4a0a066d3e0831aeed8b94cd828b7b79ca63ca5a381a17a5c3f78"}
May 26 12:04:46.841135 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.840843    2563 scope.go:117] "RemoveContainer" containerID="bd362f09904ccbcabdf9cf2c34804f33242155a58cc0ff186582e29898d6ac64"
May 26 12:04:46.849443 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.849424    2563 scope.go:117] "RemoveContainer" containerID="f279a87b11993b1d8a080817a28d303d302b2d4045a26186b13fa80ddbc5877a"
May 26 12:04:46.856399 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.856383    2563 scope.go:117] "RemoveContainer" containerID="bd362f09904ccbcabdf9cf2c34804f33242155a58cc0ff186582e29898d6ac64"
May 26 12:04:46.856630 ip-10-0-135-71 kubenswrapper[2563]: E0526 12:04:46.856610    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd362f09904ccbcabdf9cf2c34804f33242155a58cc0ff186582e29898d6ac64\": container with ID starting with bd362f09904ccbcabdf9cf2c34804f33242155a58cc0ff186582e29898d6ac64 not found: ID does not exist" containerID="bd362f09904ccbcabdf9cf2c34804f33242155a58cc0ff186582e29898d6ac64"
May 26 12:04:46.856691 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.856637    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd362f09904ccbcabdf9cf2c34804f33242155a58cc0ff186582e29898d6ac64"} err="failed to get container status \"bd362f09904ccbcabdf9cf2c34804f33242155a58cc0ff186582e29898d6ac64\": rpc error: code = NotFound desc = could not find container \"bd362f09904ccbcabdf9cf2c34804f33242155a58cc0ff186582e29898d6ac64\": container with ID starting with bd362f09904ccbcabdf9cf2c34804f33242155a58cc0ff186582e29898d6ac64 not found: ID does not exist"
May 26 12:04:46.856691 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.856654    2563 scope.go:117] "RemoveContainer" containerID="f279a87b11993b1d8a080817a28d303d302b2d4045a26186b13fa80ddbc5877a"
May 26 12:04:46.856870 ip-10-0-135-71 kubenswrapper[2563]: E0526 12:04:46.856852    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f279a87b11993b1d8a080817a28d303d302b2d4045a26186b13fa80ddbc5877a\": container with ID starting with f279a87b11993b1d8a080817a28d303d302b2d4045a26186b13fa80ddbc5877a not found: ID does not exist" containerID="f279a87b11993b1d8a080817a28d303d302b2d4045a26186b13fa80ddbc5877a"
May 26 12:04:46.856917 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.856875    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f279a87b11993b1d8a080817a28d303d302b2d4045a26186b13fa80ddbc5877a"} err="failed to get container status \"f279a87b11993b1d8a080817a28d303d302b2d4045a26186b13fa80ddbc5877a\": rpc error: code = NotFound desc = could not find container \"f279a87b11993b1d8a080817a28d303d302b2d4045a26186b13fa80ddbc5877a\": container with ID starting with f279a87b11993b1d8a080817a28d303d302b2d4045a26186b13fa80ddbc5877a not found: ID does not exist"
May 26 12:04:46.865738 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.865709    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"]
May 26 12:04:46.871483 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:46.871458    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c458-predictor-7fc8474d4d-jdr7v"]
May 26 12:04:47.796869 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:47.796834    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" path="/var/lib/kubelet/pods/65e63c61-bd7f-4e6b-b2f8-779a68944c37/volumes"
May 26 12:04:50.839805 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:50.839776    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:04:50.840298 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:50.840249    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks" podUID="6fc22dc1-5779-4102-9260-8107f828eddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 26 12:04:51.721643 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:04:51.721599    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc" podUID="25b5fa81-3e00-4738-b260-086600720195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 26 12:05:00.841006 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:00.840961    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks" podUID="6fc22dc1-5779-4102-9260-8107f828eddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 26 12:05:01.722007 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:01.721976    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:05:10.841189 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:10.841143    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks" podUID="6fc22dc1-5779-4102-9260-8107f828eddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 26 12:05:20.840557 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:20.840511    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks" podUID="6fc22dc1-5779-4102-9260-8107f828eddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 26 12:05:23.637062 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.637027    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"]
May 26 12:05:23.637618 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.637389    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc" podUID="25b5fa81-3e00-4738-b260-086600720195" containerName="kserve-container" containerID="cri-o://a1f393eebf4b21540ed949cc82d83eab589ac772ccca63c0a9ddc473f4f14d53" gracePeriod=30
May 26 12:05:23.637618 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.637480    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc" podUID="25b5fa81-3e00-4738-b260-086600720195" containerName="kube-rbac-proxy" containerID="cri-o://228e78bbf649b19afb2669a2c375df1fc9162c430ce23ca974578fb074061e8f" gracePeriod=30
May 26 12:05:23.668020 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.667995    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"]
May 26 12:05:23.668614 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.668586    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" containerName="kube-rbac-proxy"
May 26 12:05:23.668614 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.668611    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" containerName="kube-rbac-proxy"
May 26 12:05:23.668734 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.668650    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" containerName="kserve-container"
May 26 12:05:23.668734 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.668657    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" containerName="kserve-container"
May 26 12:05:23.668813 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.668740    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" containerName="kserve-container"
May 26 12:05:23.668813 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.668751    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="65e63c61-bd7f-4e6b-b2f8-779a68944c37" containerName="kube-rbac-proxy"
May 26 12:05:23.672151 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.672132    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:05:23.680770 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.680403    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-52624-predictor-serving-cert\""
May 26 12:05:23.680770 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.680760    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-52624-kube-rbac-proxy-sar-config\""
May 26 12:05:23.686667 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.686636    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"]
May 26 12:05:23.817849 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.817800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-52624-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4930a12-156e-40d8-9306-5b0598b02c9c-success-200-isvc-52624-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-52624-predictor-68f4785b48-tj56s\" (UID: \"b4930a12-156e-40d8-9306-5b0598b02c9c\") " pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:05:23.818028 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.817908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4930a12-156e-40d8-9306-5b0598b02c9c-proxy-tls\") pod \"success-200-isvc-52624-predictor-68f4785b48-tj56s\" (UID: \"b4930a12-156e-40d8-9306-5b0598b02c9c\") " pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:05:23.818028 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.817968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg8wc\" (UniqueName: \"kubernetes.io/projected/b4930a12-156e-40d8-9306-5b0598b02c9c-kube-api-access-mg8wc\") pod \"success-200-isvc-52624-predictor-68f4785b48-tj56s\" (UID: \"b4930a12-156e-40d8-9306-5b0598b02c9c\") " pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:05:23.919589 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.919490    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-52624-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4930a12-156e-40d8-9306-5b0598b02c9c-success-200-isvc-52624-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-52624-predictor-68f4785b48-tj56s\" (UID: \"b4930a12-156e-40d8-9306-5b0598b02c9c\") " pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:05:23.919589 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.919584    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4930a12-156e-40d8-9306-5b0598b02c9c-proxy-tls\") pod \"success-200-isvc-52624-predictor-68f4785b48-tj56s\" (UID: \"b4930a12-156e-40d8-9306-5b0598b02c9c\") " pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:05:23.919828 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.919629    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mg8wc\" (UniqueName: \"kubernetes.io/projected/b4930a12-156e-40d8-9306-5b0598b02c9c-kube-api-access-mg8wc\") pod \"success-200-isvc-52624-predictor-68f4785b48-tj56s\" (UID: \"b4930a12-156e-40d8-9306-5b0598b02c9c\") " pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:05:23.919828 ip-10-0-135-71 kubenswrapper[2563]: E0526 12:05:23.919815    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-52624-predictor-serving-cert: secret "success-200-isvc-52624-predictor-serving-cert" not found
May 26 12:05:23.920009 ip-10-0-135-71 kubenswrapper[2563]: E0526 12:05:23.919905    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4930a12-156e-40d8-9306-5b0598b02c9c-proxy-tls podName:b4930a12-156e-40d8-9306-5b0598b02c9c nodeName:}" failed. No retries permitted until 2026-05-26 12:05:24.419882619 +0000 UTC m=+2457.225721072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b4930a12-156e-40d8-9306-5b0598b02c9c-proxy-tls") pod "success-200-isvc-52624-predictor-68f4785b48-tj56s" (UID: "b4930a12-156e-40d8-9306-5b0598b02c9c") : secret "success-200-isvc-52624-predictor-serving-cert" not found
May 26 12:05:23.920298 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.920259    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-52624-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4930a12-156e-40d8-9306-5b0598b02c9c-success-200-isvc-52624-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-52624-predictor-68f4785b48-tj56s\" (UID: \"b4930a12-156e-40d8-9306-5b0598b02c9c\") " pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:05:23.931664 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.931626    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg8wc\" (UniqueName: \"kubernetes.io/projected/b4930a12-156e-40d8-9306-5b0598b02c9c-kube-api-access-mg8wc\") pod \"success-200-isvc-52624-predictor-68f4785b48-tj56s\" (UID: \"b4930a12-156e-40d8-9306-5b0598b02c9c\") " pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:05:23.965111 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.965081    2563 generic.go:358] "Generic (PLEG): container finished" podID="25b5fa81-3e00-4738-b260-086600720195" containerID="228e78bbf649b19afb2669a2c375df1fc9162c430ce23ca974578fb074061e8f" exitCode=2
May 26 12:05:23.965263 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:23.965147    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc" event={"ID":"25b5fa81-3e00-4738-b260-086600720195","Type":"ContainerDied","Data":"228e78bbf649b19afb2669a2c375df1fc9162c430ce23ca974578fb074061e8f"}
May 26 12:05:24.422686 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:24.422649    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4930a12-156e-40d8-9306-5b0598b02c9c-proxy-tls\") pod \"success-200-isvc-52624-predictor-68f4785b48-tj56s\" (UID: \"b4930a12-156e-40d8-9306-5b0598b02c9c\") " pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:05:24.425003 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:24.424978    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4930a12-156e-40d8-9306-5b0598b02c9c-proxy-tls\") pod \"success-200-isvc-52624-predictor-68f4785b48-tj56s\" (UID: \"b4930a12-156e-40d8-9306-5b0598b02c9c\") " pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:05:24.595569 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:24.595533    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:05:24.724193 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:24.724159    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"]
May 26 12:05:24.727322 ip-10-0-135-71 kubenswrapper[2563]: W0526 12:05:24.727297    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4930a12_156e_40d8_9306_5b0598b02c9c.slice/crio-71f698104e7e2cb9b29ccfb629eb538f0466025c1d8a6bcdf30448353a621c9d WatchSource:0}: Error finding container 71f698104e7e2cb9b29ccfb629eb538f0466025c1d8a6bcdf30448353a621c9d: Status 404 returned error can't find the container with id 71f698104e7e2cb9b29ccfb629eb538f0466025c1d8a6bcdf30448353a621c9d
May 26 12:05:24.970766 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:24.970679    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" event={"ID":"b4930a12-156e-40d8-9306-5b0598b02c9c","Type":"ContainerStarted","Data":"d5bb3b0389d6f08f4f70bf42dac3a47ac08cfa9e9cec0b52ff1fdd1ab4623286"}
May 26 12:05:24.970766 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:24.970721    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" event={"ID":"b4930a12-156e-40d8-9306-5b0598b02c9c","Type":"ContainerStarted","Data":"fca135b15a7c654a6240c8cb6580b2ed477f825ddd883a378963f07f9b8ea077"}
May 26 12:05:24.970766 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:24.970737    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" event={"ID":"b4930a12-156e-40d8-9306-5b0598b02c9c","Type":"ContainerStarted","Data":"71f698104e7e2cb9b29ccfb629eb538f0466025c1d8a6bcdf30448353a621c9d"}
May 26 12:05:24.970977 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:24.970831    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:05:24.970977 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:24.970851    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:05:24.972198 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:24.972169    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
May 26 12:05:24.991412 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:24.991364    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" podStartSLOduration=1.9913499890000002 podStartE2EDuration="1.991349989s" podCreationTimestamp="2026-05-26 12:05:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:05:24.989534231 +0000 UTC m=+2457.795372695" watchObservedRunningTime="2026-05-26 12:05:24.991349989 +0000 UTC m=+2457.797188452"
May 26 12:05:25.974258 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:25.974214    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
May 26 12:05:26.778484 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.778462    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:05:26.944187 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.944097    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85hrr\" (UniqueName: \"kubernetes.io/projected/25b5fa81-3e00-4738-b260-086600720195-kube-api-access-85hrr\") pod \"25b5fa81-3e00-4738-b260-086600720195\" (UID: \"25b5fa81-3e00-4738-b260-086600720195\") "
May 26 12:05:26.944349 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.944192    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-81833-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25b5fa81-3e00-4738-b260-086600720195-success-200-isvc-81833-kube-rbac-proxy-sar-config\") pod \"25b5fa81-3e00-4738-b260-086600720195\" (UID: \"25b5fa81-3e00-4738-b260-086600720195\") "
May 26 12:05:26.944349 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.944217    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25b5fa81-3e00-4738-b260-086600720195-proxy-tls\") pod \"25b5fa81-3e00-4738-b260-086600720195\" (UID: \"25b5fa81-3e00-4738-b260-086600720195\") "
May 26 12:05:26.944596 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.944572    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b5fa81-3e00-4738-b260-086600720195-success-200-isvc-81833-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-81833-kube-rbac-proxy-sar-config") pod "25b5fa81-3e00-4738-b260-086600720195" (UID: "25b5fa81-3e00-4738-b260-086600720195"). InnerVolumeSpecName "success-200-isvc-81833-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:05:26.946307 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.946264    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b5fa81-3e00-4738-b260-086600720195-kube-api-access-85hrr" (OuterVolumeSpecName: "kube-api-access-85hrr") pod "25b5fa81-3e00-4738-b260-086600720195" (UID: "25b5fa81-3e00-4738-b260-086600720195"). InnerVolumeSpecName "kube-api-access-85hrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:05:26.946427 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.946313    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b5fa81-3e00-4738-b260-086600720195-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "25b5fa81-3e00-4738-b260-086600720195" (UID: "25b5fa81-3e00-4738-b260-086600720195"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:05:26.978907 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.978875    2563 generic.go:358] "Generic (PLEG): container finished" podID="25b5fa81-3e00-4738-b260-086600720195" containerID="a1f393eebf4b21540ed949cc82d83eab589ac772ccca63c0a9ddc473f4f14d53" exitCode=0
May 26 12:05:26.979225 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.978948    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"
May 26 12:05:26.979225 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.978952    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc" event={"ID":"25b5fa81-3e00-4738-b260-086600720195","Type":"ContainerDied","Data":"a1f393eebf4b21540ed949cc82d83eab589ac772ccca63c0a9ddc473f4f14d53"}
May 26 12:05:26.979225 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.978992    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc" event={"ID":"25b5fa81-3e00-4738-b260-086600720195","Type":"ContainerDied","Data":"68c40ab5f805049dbf420711386376c45dc73f56ec03ed01070f5d5f474fd093"}
May 26 12:05:26.979225 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.979011    2563 scope.go:117] "RemoveContainer" containerID="228e78bbf649b19afb2669a2c375df1fc9162c430ce23ca974578fb074061e8f"
May 26 12:05:26.988151 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.987952    2563 scope.go:117] "RemoveContainer" containerID="a1f393eebf4b21540ed949cc82d83eab589ac772ccca63c0a9ddc473f4f14d53"
May 26 12:05:26.996139 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.996113    2563 scope.go:117] "RemoveContainer" containerID="228e78bbf649b19afb2669a2c375df1fc9162c430ce23ca974578fb074061e8f"
May 26 12:05:26.996470 ip-10-0-135-71 kubenswrapper[2563]: E0526 12:05:26.996450    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228e78bbf649b19afb2669a2c375df1fc9162c430ce23ca974578fb074061e8f\": container with ID starting with 228e78bbf649b19afb2669a2c375df1fc9162c430ce23ca974578fb074061e8f not found: ID does not exist" containerID="228e78bbf649b19afb2669a2c375df1fc9162c430ce23ca974578fb074061e8f"
May 26 12:05:26.996530 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.996481    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228e78bbf649b19afb2669a2c375df1fc9162c430ce23ca974578fb074061e8f"} err="failed to get container status \"228e78bbf649b19afb2669a2c375df1fc9162c430ce23ca974578fb074061e8f\": rpc error: code = NotFound desc = could not find container \"228e78bbf649b19afb2669a2c375df1fc9162c430ce23ca974578fb074061e8f\": container with ID starting with 228e78bbf649b19afb2669a2c375df1fc9162c430ce23ca974578fb074061e8f not found: ID does not exist"
May 26 12:05:26.996530 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.996500    2563 scope.go:117] "RemoveContainer" containerID="a1f393eebf4b21540ed949cc82d83eab589ac772ccca63c0a9ddc473f4f14d53"
May 26 12:05:26.996769 ip-10-0-135-71 kubenswrapper[2563]: E0526 12:05:26.996748    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f393eebf4b21540ed949cc82d83eab589ac772ccca63c0a9ddc473f4f14d53\": container with ID starting with a1f393eebf4b21540ed949cc82d83eab589ac772ccca63c0a9ddc473f4f14d53 not found: ID does not exist" containerID="a1f393eebf4b21540ed949cc82d83eab589ac772ccca63c0a9ddc473f4f14d53"
May 26 12:05:26.996823 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:26.996779    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f393eebf4b21540ed949cc82d83eab589ac772ccca63c0a9ddc473f4f14d53"} err="failed to get container status \"a1f393eebf4b21540ed949cc82d83eab589ac772ccca63c0a9ddc473f4f14d53\": rpc error: code = NotFound desc = could not find container \"a1f393eebf4b21540ed949cc82d83eab589ac772ccca63c0a9ddc473f4f14d53\": container with ID starting with a1f393eebf4b21540ed949cc82d83eab589ac772ccca63c0a9ddc473f4f14d53 not found: ID does not exist"
May 26 12:05:27.004782 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:27.004757    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"]
May 26 12:05:27.009246 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:27.009222    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc"]
May 26 12:05:27.045681 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:27.045650    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85hrr\" (UniqueName: \"kubernetes.io/projected/25b5fa81-3e00-4738-b260-086600720195-kube-api-access-85hrr\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 12:05:27.045681 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:27.045676    2563 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-81833-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25b5fa81-3e00-4738-b260-086600720195-success-200-isvc-81833-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 12:05:27.045681 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:27.045688    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25b5fa81-3e00-4738-b260-086600720195-proxy-tls\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 12:05:27.716900 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:27.716853    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-81833-predictor-5554ddb76-qccpc" podUID="25b5fa81-3e00-4738-b260-086600720195" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": context deadline exceeded"
May 26 12:05:27.796970 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:27.796936    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25b5fa81-3e00-4738-b260-086600720195" path="/var/lib/kubelet/pods/25b5fa81-3e00-4738-b260-086600720195/volumes"
May 26 12:05:30.841297 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:30.841251    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:05:30.979355 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:30.979327    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:05:30.979838 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:30.979811    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
May 26 12:05:40.979806 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:40.979764    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
May 26 12:05:50.980016 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:05:50.979974    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
May 26 12:06:00.980050 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:06:00.980006    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
May 26 12:06:10.980511 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:06:10.980427    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:14:38.530766 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:38.530733    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"]
May 26 12:14:38.533271 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:38.531013    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerName="kserve-container" containerID="cri-o://fca135b15a7c654a6240c8cb6580b2ed477f825ddd883a378963f07f9b8ea077" gracePeriod=30
May 26 12:14:38.533271 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:38.531060    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerName="kube-rbac-proxy" containerID="cri-o://d5bb3b0389d6f08f4f70bf42dac3a47ac08cfa9e9cec0b52ff1fdd1ab4623286" gracePeriod=30
May 26 12:14:38.895120 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:38.895030    2563 generic.go:358] "Generic (PLEG): container finished" podID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerID="d5bb3b0389d6f08f4f70bf42dac3a47ac08cfa9e9cec0b52ff1fdd1ab4623286" exitCode=2
May 26 12:14:38.895120 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:38.895098    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" event={"ID":"b4930a12-156e-40d8-9306-5b0598b02c9c","Type":"ContainerDied","Data":"d5bb3b0389d6f08f4f70bf42dac3a47ac08cfa9e9cec0b52ff1fdd1ab4623286"}
May 26 12:14:40.975275 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:40.975220    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
May 26 12:14:40.979792 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:40.979753    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
May 26 12:14:41.685929 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.685904    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:14:41.768638 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.768597    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4930a12-156e-40d8-9306-5b0598b02c9c-proxy-tls\") pod \"b4930a12-156e-40d8-9306-5b0598b02c9c\" (UID: \"b4930a12-156e-40d8-9306-5b0598b02c9c\") "
May 26 12:14:41.768821 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.768707    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg8wc\" (UniqueName: \"kubernetes.io/projected/b4930a12-156e-40d8-9306-5b0598b02c9c-kube-api-access-mg8wc\") pod \"b4930a12-156e-40d8-9306-5b0598b02c9c\" (UID: \"b4930a12-156e-40d8-9306-5b0598b02c9c\") "
May 26 12:14:41.768821 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.768778    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-52624-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4930a12-156e-40d8-9306-5b0598b02c9c-success-200-isvc-52624-kube-rbac-proxy-sar-config\") pod \"b4930a12-156e-40d8-9306-5b0598b02c9c\" (UID: \"b4930a12-156e-40d8-9306-5b0598b02c9c\") "
May 26 12:14:41.769154 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.769121    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4930a12-156e-40d8-9306-5b0598b02c9c-success-200-isvc-52624-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-52624-kube-rbac-proxy-sar-config") pod "b4930a12-156e-40d8-9306-5b0598b02c9c" (UID: "b4930a12-156e-40d8-9306-5b0598b02c9c"). InnerVolumeSpecName "success-200-isvc-52624-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:14:41.770756 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.770721    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4930a12-156e-40d8-9306-5b0598b02c9c-kube-api-access-mg8wc" (OuterVolumeSpecName: "kube-api-access-mg8wc") pod "b4930a12-156e-40d8-9306-5b0598b02c9c" (UID: "b4930a12-156e-40d8-9306-5b0598b02c9c"). InnerVolumeSpecName "kube-api-access-mg8wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:14:41.770756 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.770721    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4930a12-156e-40d8-9306-5b0598b02c9c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b4930a12-156e-40d8-9306-5b0598b02c9c" (UID: "b4930a12-156e-40d8-9306-5b0598b02c9c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:14:41.869565 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.869530    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4930a12-156e-40d8-9306-5b0598b02c9c-proxy-tls\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 12:14:41.869565 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.869564    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg8wc\" (UniqueName: \"kubernetes.io/projected/b4930a12-156e-40d8-9306-5b0598b02c9c-kube-api-access-mg8wc\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 12:14:41.869766 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.869578    2563 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-52624-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4930a12-156e-40d8-9306-5b0598b02c9c-success-200-isvc-52624-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 12:14:41.907068 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.907026    2563 generic.go:358] "Generic (PLEG): container finished" podID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerID="fca135b15a7c654a6240c8cb6580b2ed477f825ddd883a378963f07f9b8ea077" exitCode=0
May 26 12:14:41.907220 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.907106    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"
May 26 12:14:41.907220 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.907113    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" event={"ID":"b4930a12-156e-40d8-9306-5b0598b02c9c","Type":"ContainerDied","Data":"fca135b15a7c654a6240c8cb6580b2ed477f825ddd883a378963f07f9b8ea077"}
May 26 12:14:41.907220 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.907153    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s" event={"ID":"b4930a12-156e-40d8-9306-5b0598b02c9c","Type":"ContainerDied","Data":"71f698104e7e2cb9b29ccfb629eb538f0466025c1d8a6bcdf30448353a621c9d"}
May 26 12:14:41.907220 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.907168    2563 scope.go:117] "RemoveContainer" containerID="d5bb3b0389d6f08f4f70bf42dac3a47ac08cfa9e9cec0b52ff1fdd1ab4623286"
May 26 12:14:41.915433 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.915416    2563 scope.go:117] "RemoveContainer" containerID="fca135b15a7c654a6240c8cb6580b2ed477f825ddd883a378963f07f9b8ea077"
May 26 12:14:41.922486 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.922451    2563 scope.go:117] "RemoveContainer" containerID="d5bb3b0389d6f08f4f70bf42dac3a47ac08cfa9e9cec0b52ff1fdd1ab4623286"
May 26 12:14:41.923121 ip-10-0-135-71 kubenswrapper[2563]: E0526 12:14:41.923090    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5bb3b0389d6f08f4f70bf42dac3a47ac08cfa9e9cec0b52ff1fdd1ab4623286\": container with ID starting with d5bb3b0389d6f08f4f70bf42dac3a47ac08cfa9e9cec0b52ff1fdd1ab4623286 not found: ID does not exist" containerID="d5bb3b0389d6f08f4f70bf42dac3a47ac08cfa9e9cec0b52ff1fdd1ab4623286"
May 26 12:14:41.923228 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.923131    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5bb3b0389d6f08f4f70bf42dac3a47ac08cfa9e9cec0b52ff1fdd1ab4623286"} err="failed to get container status \"d5bb3b0389d6f08f4f70bf42dac3a47ac08cfa9e9cec0b52ff1fdd1ab4623286\": rpc error: code = NotFound desc = could not find container \"d5bb3b0389d6f08f4f70bf42dac3a47ac08cfa9e9cec0b52ff1fdd1ab4623286\": container with ID starting with d5bb3b0389d6f08f4f70bf42dac3a47ac08cfa9e9cec0b52ff1fdd1ab4623286 not found: ID does not exist"
May 26 12:14:41.923228 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.923186    2563 scope.go:117] "RemoveContainer" containerID="fca135b15a7c654a6240c8cb6580b2ed477f825ddd883a378963f07f9b8ea077"
May 26 12:14:41.923544 ip-10-0-135-71 kubenswrapper[2563]: E0526 12:14:41.923515    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca135b15a7c654a6240c8cb6580b2ed477f825ddd883a378963f07f9b8ea077\": container with ID starting with fca135b15a7c654a6240c8cb6580b2ed477f825ddd883a378963f07f9b8ea077 not found: ID does not exist" containerID="fca135b15a7c654a6240c8cb6580b2ed477f825ddd883a378963f07f9b8ea077"
May 26 12:14:41.923632 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.923553    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca135b15a7c654a6240c8cb6580b2ed477f825ddd883a378963f07f9b8ea077"} err="failed to get container status \"fca135b15a7c654a6240c8cb6580b2ed477f825ddd883a378963f07f9b8ea077\": rpc error: code = NotFound desc = could not find container \"fca135b15a7c654a6240c8cb6580b2ed477f825ddd883a378963f07f9b8ea077\": container with ID starting with fca135b15a7c654a6240c8cb6580b2ed477f825ddd883a378963f07f9b8ea077 not found: ID does not exist"
May 26 12:14:41.924868 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.924850    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"]
May 26 12:14:41.932094 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:41.932071    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-52624-predictor-68f4785b48-tj56s"]
May 26 12:14:43.796199 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:14:43.796165    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" path="/var/lib/kubelet/pods/b4930a12-156e-40d8-9306-5b0598b02c9c/volumes"
May 26 12:22:02.547817 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:02.547777    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"]
May 26 12:22:02.550219 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:02.548069    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks" podUID="6fc22dc1-5779-4102-9260-8107f828eddb" containerName="kserve-container" containerID="cri-o://ddf94291a8d830e947ea31e52ba3dfa54497201f84cee4621073592a9bc52eb3" gracePeriod=30
May 26 12:22:02.550219 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:02.548124    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks" podUID="6fc22dc1-5779-4102-9260-8107f828eddb" containerName="kube-rbac-proxy" containerID="cri-o://b164e838e909c834747e05319d58de1f85af5aca52de972c5dcd40fa072b6c90" gracePeriod=30
May 26 12:22:03.440724 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:03.440689    2563 generic.go:358] "Generic (PLEG): container finished" podID="6fc22dc1-5779-4102-9260-8107f828eddb" containerID="b164e838e909c834747e05319d58de1f85af5aca52de972c5dcd40fa072b6c90" exitCode=2
May 26 12:22:03.440908 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:03.440757    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks" event={"ID":"6fc22dc1-5779-4102-9260-8107f828eddb","Type":"ContainerDied","Data":"b164e838e909c834747e05319d58de1f85af5aca52de972c5dcd40fa072b6c90"}
May 26 12:22:05.688222 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:05.688196    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:22:05.719822 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:05.719793    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfr5x\" (UniqueName: \"kubernetes.io/projected/6fc22dc1-5779-4102-9260-8107f828eddb-kube-api-access-gfr5x\") pod \"6fc22dc1-5779-4102-9260-8107f828eddb\" (UID: \"6fc22dc1-5779-4102-9260-8107f828eddb\") "
May 26 12:22:05.719977 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:05.719856    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fc22dc1-5779-4102-9260-8107f828eddb-proxy-tls\") pod \"6fc22dc1-5779-4102-9260-8107f828eddb\" (UID: \"6fc22dc1-5779-4102-9260-8107f828eddb\") "
May 26 12:22:05.719977 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:05.719908    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-152bc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fc22dc1-5779-4102-9260-8107f828eddb-success-200-isvc-152bc-kube-rbac-proxy-sar-config\") pod \"6fc22dc1-5779-4102-9260-8107f828eddb\" (UID: \"6fc22dc1-5779-4102-9260-8107f828eddb\") "
May 26 12:22:05.720333 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:05.720277    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc22dc1-5779-4102-9260-8107f828eddb-success-200-isvc-152bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-152bc-kube-rbac-proxy-sar-config") pod "6fc22dc1-5779-4102-9260-8107f828eddb" (UID: "6fc22dc1-5779-4102-9260-8107f828eddb"). InnerVolumeSpecName "success-200-isvc-152bc-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:22:05.721827 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:05.721805    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc22dc1-5779-4102-9260-8107f828eddb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6fc22dc1-5779-4102-9260-8107f828eddb" (UID: "6fc22dc1-5779-4102-9260-8107f828eddb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:22:05.721909 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:05.721886    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc22dc1-5779-4102-9260-8107f828eddb-kube-api-access-gfr5x" (OuterVolumeSpecName: "kube-api-access-gfr5x") pod "6fc22dc1-5779-4102-9260-8107f828eddb" (UID: "6fc22dc1-5779-4102-9260-8107f828eddb"). InnerVolumeSpecName "kube-api-access-gfr5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:22:05.820474 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:05.820443    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfr5x\" (UniqueName: \"kubernetes.io/projected/6fc22dc1-5779-4102-9260-8107f828eddb-kube-api-access-gfr5x\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 12:22:05.820474 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:05.820469    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fc22dc1-5779-4102-9260-8107f828eddb-proxy-tls\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 12:22:05.820674 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:05.820480    2563 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-152bc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fc22dc1-5779-4102-9260-8107f828eddb-success-200-isvc-152bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-71.ec2.internal\" DevicePath \"\""
May 26 12:22:06.451865 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:06.451820    2563 generic.go:358] "Generic (PLEG): container finished" podID="6fc22dc1-5779-4102-9260-8107f828eddb" containerID="ddf94291a8d830e947ea31e52ba3dfa54497201f84cee4621073592a9bc52eb3" exitCode=0
May 26 12:22:06.452075 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:06.451912    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks" event={"ID":"6fc22dc1-5779-4102-9260-8107f828eddb","Type":"ContainerDied","Data":"ddf94291a8d830e947ea31e52ba3dfa54497201f84cee4621073592a9bc52eb3"}
May 26 12:22:06.452075 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:06.451925    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"
May 26 12:22:06.452075 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:06.451957    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks" event={"ID":"6fc22dc1-5779-4102-9260-8107f828eddb","Type":"ContainerDied","Data":"35e479b18ee452b3fb2e19372babc8258b5aae06b83aa2b7602e33a420c9bb11"}
May 26 12:22:06.452075 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:06.451974    2563 scope.go:117] "RemoveContainer" containerID="b164e838e909c834747e05319d58de1f85af5aca52de972c5dcd40fa072b6c90"
May 26 12:22:06.461483 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:06.461464    2563 scope.go:117] "RemoveContainer" containerID="ddf94291a8d830e947ea31e52ba3dfa54497201f84cee4621073592a9bc52eb3"
May 26 12:22:06.468832 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:06.468816    2563 scope.go:117] "RemoveContainer" containerID="b164e838e909c834747e05319d58de1f85af5aca52de972c5dcd40fa072b6c90"
May 26 12:22:06.469072 ip-10-0-135-71 kubenswrapper[2563]: E0526 12:22:06.469050    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b164e838e909c834747e05319d58de1f85af5aca52de972c5dcd40fa072b6c90\": container with ID starting with b164e838e909c834747e05319d58de1f85af5aca52de972c5dcd40fa072b6c90 not found: ID does not exist" containerID="b164e838e909c834747e05319d58de1f85af5aca52de972c5dcd40fa072b6c90"
May 26 12:22:06.469145 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:06.469084    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b164e838e909c834747e05319d58de1f85af5aca52de972c5dcd40fa072b6c90"} err="failed to get container status \"b164e838e909c834747e05319d58de1f85af5aca52de972c5dcd40fa072b6c90\": rpc error: code = NotFound desc = could not find container \"b164e838e909c834747e05319d58de1f85af5aca52de972c5dcd40fa072b6c90\": container with ID starting with b164e838e909c834747e05319d58de1f85af5aca52de972c5dcd40fa072b6c90 not found: ID does not exist"
May 26 12:22:06.469145 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:06.469107    2563 scope.go:117] "RemoveContainer" containerID="ddf94291a8d830e947ea31e52ba3dfa54497201f84cee4621073592a9bc52eb3"
May 26 12:22:06.469363 ip-10-0-135-71 kubenswrapper[2563]: E0526 12:22:06.469342    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf94291a8d830e947ea31e52ba3dfa54497201f84cee4621073592a9bc52eb3\": container with ID starting with ddf94291a8d830e947ea31e52ba3dfa54497201f84cee4621073592a9bc52eb3 not found: ID does not exist" containerID="ddf94291a8d830e947ea31e52ba3dfa54497201f84cee4621073592a9bc52eb3"
May 26 12:22:06.469465 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:06.469366    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf94291a8d830e947ea31e52ba3dfa54497201f84cee4621073592a9bc52eb3"} err="failed to get container status \"ddf94291a8d830e947ea31e52ba3dfa54497201f84cee4621073592a9bc52eb3\": rpc error: code = NotFound desc = could not find container \"ddf94291a8d830e947ea31e52ba3dfa54497201f84cee4621073592a9bc52eb3\": container with ID starting with ddf94291a8d830e947ea31e52ba3dfa54497201f84cee4621073592a9bc52eb3 not found: ID does not exist"
May 26 12:22:06.495338 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:06.495306    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"]
May 26 12:22:06.514780 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:06.514755    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-152bc-predictor-7456d7b59b-hxqks"]
May 26 12:22:07.795819 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:07.795785    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fc22dc1-5779-4102-9260-8107f828eddb" path="/var/lib/kubelet/pods/6fc22dc1-5779-4102-9260-8107f828eddb/volumes"
May 26 12:22:36.807180 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:36.807133    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wzshh_7f39775e-7a53-44fb-8e18-70fd6a5764c5/global-pull-secret-syncer/0.log"
May 26 12:22:36.954894 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:36.954863    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zk6bl_bd8ad6b4-eb8d-41fe-8247-de669518e6d4/konnectivity-agent/0.log"
May 26 12:22:37.074620 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:37.074503    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-71.ec2.internal_9a50d4f711fbf255cb7babbc8960a87b/haproxy/0.log"
May 26 12:22:40.353375 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:40.353298    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_893717db-1c0f-4a5c-b9d0-b0fb328039e6/alertmanager/0.log"
May 26 12:22:40.376583 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:40.376560    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_893717db-1c0f-4a5c-b9d0-b0fb328039e6/config-reloader/0.log"
May 26 12:22:40.401858 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:40.401832    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_893717db-1c0f-4a5c-b9d0-b0fb328039e6/kube-rbac-proxy-web/0.log"
May 26 12:22:40.423765 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:40.423741    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_893717db-1c0f-4a5c-b9d0-b0fb328039e6/kube-rbac-proxy/0.log"
May 26 12:22:40.447709 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:40.447684    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_893717db-1c0f-4a5c-b9d0-b0fb328039e6/kube-rbac-proxy-metric/0.log"
May 26 12:22:40.477678 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:40.477647    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_893717db-1c0f-4a5c-b9d0-b0fb328039e6/prom-label-proxy/0.log"
May 26 12:22:40.504172 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:40.504145    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_893717db-1c0f-4a5c-b9d0-b0fb328039e6/init-config-reloader/0.log"
May 26 12:22:40.573039 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:40.573015    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-277l9_37858259-5496-44a7-984d-78eafc49f221/kube-state-metrics/0.log"
May 26 12:22:40.597076 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:40.597055    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-277l9_37858259-5496-44a7-984d-78eafc49f221/kube-rbac-proxy-main/0.log"
May 26 12:22:40.618340 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:40.618256    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-277l9_37858259-5496-44a7-984d-78eafc49f221/kube-rbac-proxy-self/0.log"
May 26 12:22:40.722216 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:40.722187    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-949ct_486de637-3719-41d0-bc8a-d1969384daa3/node-exporter/0.log"
May 26 12:22:40.755060 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:40.755033    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-949ct_486de637-3719-41d0-bc8a-d1969384daa3/kube-rbac-proxy/0.log"
May 26 12:22:40.785555 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:40.785525    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-949ct_486de637-3719-41d0-bc8a-d1969384daa3/init-textfile/0.log"
May 26 12:22:40.966630 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:40.966557    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-dbfkl_93f480d3-f56c-4fa5-9ad4-dd2252a70d5a/kube-rbac-proxy-main/0.log"
May 26 12:22:40.990574 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:40.990546    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-dbfkl_93f480d3-f56c-4fa5-9ad4-dd2252a70d5a/kube-rbac-proxy-self/0.log"
May 26 12:22:41.013468 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:41.013441    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-dbfkl_93f480d3-f56c-4fa5-9ad4-dd2252a70d5a/openshift-state-metrics/0.log"
May 26 12:22:41.269901 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:41.269872    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-p4cml_df896a28-f4ca-4dd4-bd50-39894600be5d/prometheus-operator-admission-webhook/0.log"
May 26 12:22:41.305706 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:41.305680    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bf6bb686f-67pvg_c4256857-00c8-400a-b4bf-b58d10ed11c9/telemeter-client/0.log"
May 26 12:22:41.331072 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:41.331046    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bf6bb686f-67pvg_c4256857-00c8-400a-b4bf-b58d10ed11c9/reload/0.log"
May 26 12:22:41.355720 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:41.355694    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bf6bb686f-67pvg_c4256857-00c8-400a-b4bf-b58d10ed11c9/kube-rbac-proxy/0.log"
May 26 12:22:41.389221 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:41.389192    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f779f9ffb-4wxmj_2e736287-4f5d-4a35-b5c9-5124cffe8b06/thanos-query/0.log"
May 26 12:22:41.417093 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:41.417066    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f779f9ffb-4wxmj_2e736287-4f5d-4a35-b5c9-5124cffe8b06/kube-rbac-proxy-web/0.log"
May 26 12:22:41.438426 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:41.438399    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f779f9ffb-4wxmj_2e736287-4f5d-4a35-b5c9-5124cffe8b06/kube-rbac-proxy/0.log"
May 26 12:22:41.461167 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:41.461132    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f779f9ffb-4wxmj_2e736287-4f5d-4a35-b5c9-5124cffe8b06/prom-label-proxy/0.log"
May 26 12:22:41.482783 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:41.482756    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f779f9ffb-4wxmj_2e736287-4f5d-4a35-b5c9-5124cffe8b06/kube-rbac-proxy-rules/0.log"
May 26 12:22:41.504594 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:41.504567    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f779f9ffb-4wxmj_2e736287-4f5d-4a35-b5c9-5124cffe8b06/kube-rbac-proxy-metrics/0.log"
May 26 12:22:44.003679 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.003652    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b56c5bfb5-2btzw_500d8607-95ab-4f67-889a-66b501c166c7/console/0.log"
May 26 12:22:44.043383 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043351    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"]
May 26 12:22:44.043703 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043691    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fc22dc1-5779-4102-9260-8107f828eddb" containerName="kserve-container"
May 26 12:22:44.043752 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043706    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fc22dc1-5779-4102-9260-8107f828eddb" containerName="kserve-container"
May 26 12:22:44.043752 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043715    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fc22dc1-5779-4102-9260-8107f828eddb" containerName="kube-rbac-proxy"
May 26 12:22:44.043752 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043720    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fc22dc1-5779-4102-9260-8107f828eddb" containerName="kube-rbac-proxy"
May 26 12:22:44.043752 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043731    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerName="kube-rbac-proxy"
May 26 12:22:44.043752 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043737    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerName="kube-rbac-proxy"
May 26 12:22:44.043752 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043744    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25b5fa81-3e00-4738-b260-086600720195" containerName="kserve-container"
May 26 12:22:44.043752 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043750    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25b5fa81-3e00-4738-b260-086600720195" containerName="kserve-container"
May 26 12:22:44.043963 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043760    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerName="kserve-container"
May 26 12:22:44.043963 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043765    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerName="kserve-container"
May 26 12:22:44.043963 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043780    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25b5fa81-3e00-4738-b260-086600720195" containerName="kube-rbac-proxy"
May 26 12:22:44.043963 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043785    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25b5fa81-3e00-4738-b260-086600720195" containerName="kube-rbac-proxy"
May 26 12:22:44.043963 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043834    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerName="kube-rbac-proxy"
May 26 12:22:44.043963 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043843    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="6fc22dc1-5779-4102-9260-8107f828eddb" containerName="kserve-container"
May 26 12:22:44.043963 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043851    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="b4930a12-156e-40d8-9306-5b0598b02c9c" containerName="kserve-container"
May 26 12:22:44.043963 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043859    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="6fc22dc1-5779-4102-9260-8107f828eddb" containerName="kube-rbac-proxy"
May 26 12:22:44.043963 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043865    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="25b5fa81-3e00-4738-b260-086600720195" containerName="kserve-container"
May 26 12:22:44.043963 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.043873    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="25b5fa81-3e00-4738-b260-086600720195" containerName="kube-rbac-proxy"
May 26 12:22:44.046277 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.046261    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.055248 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.055224    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"]
May 26 12:22:44.059644 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.059617    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zm7c5\"/\"default-dockercfg-kbnnp\""
May 26 12:22:44.059852 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.059679    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zm7c5\"/\"openshift-service-ca.crt\""
May 26 12:22:44.059852 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.059717    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zm7c5\"/\"kube-root-ca.crt\""
May 26 12:22:44.153213 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.153178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3806556c-d448-4e43-b1de-e0f254f1c36e-sys\") pod \"perf-node-gather-daemonset-5srdd\" (UID: \"3806556c-d448-4e43-b1de-e0f254f1c36e\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.153403 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.153224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3806556c-d448-4e43-b1de-e0f254f1c36e-lib-modules\") pod \"perf-node-gather-daemonset-5srdd\" (UID: \"3806556c-d448-4e43-b1de-e0f254f1c36e\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.153403 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.153252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3806556c-d448-4e43-b1de-e0f254f1c36e-proc\") pod \"perf-node-gather-daemonset-5srdd\" (UID: \"3806556c-d448-4e43-b1de-e0f254f1c36e\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.153403 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.153361    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6g56\" (UniqueName: \"kubernetes.io/projected/3806556c-d448-4e43-b1de-e0f254f1c36e-kube-api-access-n6g56\") pod \"perf-node-gather-daemonset-5srdd\" (UID: \"3806556c-d448-4e43-b1de-e0f254f1c36e\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.153512 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.153449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3806556c-d448-4e43-b1de-e0f254f1c36e-podres\") pod \"perf-node-gather-daemonset-5srdd\" (UID: \"3806556c-d448-4e43-b1de-e0f254f1c36e\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.254595 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.254512    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3806556c-d448-4e43-b1de-e0f254f1c36e-sys\") pod \"perf-node-gather-daemonset-5srdd\" (UID: \"3806556c-d448-4e43-b1de-e0f254f1c36e\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.254595 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.254565    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3806556c-d448-4e43-b1de-e0f254f1c36e-lib-modules\") pod \"perf-node-gather-daemonset-5srdd\" (UID: \"3806556c-d448-4e43-b1de-e0f254f1c36e\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.254595 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.254585    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3806556c-d448-4e43-b1de-e0f254f1c36e-proc\") pod \"perf-node-gather-daemonset-5srdd\" (UID: \"3806556c-d448-4e43-b1de-e0f254f1c36e\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.254784 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.254614    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n6g56\" (UniqueName: \"kubernetes.io/projected/3806556c-d448-4e43-b1de-e0f254f1c36e-kube-api-access-n6g56\") pod \"perf-node-gather-daemonset-5srdd\" (UID: \"3806556c-d448-4e43-b1de-e0f254f1c36e\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.254784 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.254642    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3806556c-d448-4e43-b1de-e0f254f1c36e-sys\") pod \"perf-node-gather-daemonset-5srdd\" (UID: \"3806556c-d448-4e43-b1de-e0f254f1c36e\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.254784 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.254664    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3806556c-d448-4e43-b1de-e0f254f1c36e-podres\") pod \"perf-node-gather-daemonset-5srdd\" (UID: \"3806556c-d448-4e43-b1de-e0f254f1c36e\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.254784 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.254726    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3806556c-d448-4e43-b1de-e0f254f1c36e-proc\") pod \"perf-node-gather-daemonset-5srdd\" (UID: \"3806556c-d448-4e43-b1de-e0f254f1c36e\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.254784 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.254764    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3806556c-d448-4e43-b1de-e0f254f1c36e-lib-modules\") pod \"perf-node-gather-daemonset-5srdd\" (UID: \"3806556c-d448-4e43-b1de-e0f254f1c36e\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.254784 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.254782    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3806556c-d448-4e43-b1de-e0f254f1c36e-podres\") pod \"perf-node-gather-daemonset-5srdd\" (UID: \"3806556c-d448-4e43-b1de-e0f254f1c36e\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.271948 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.271911    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6g56\" (UniqueName: \"kubernetes.io/projected/3806556c-d448-4e43-b1de-e0f254f1c36e-kube-api-access-n6g56\") pod \"perf-node-gather-daemonset-5srdd\" (UID: \"3806556c-d448-4e43-b1de-e0f254f1c36e\") " pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.357601 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.357549    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:44.487991 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.487898    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"]
May 26 12:22:44.490905 ip-10-0-135-71 kubenswrapper[2563]: W0526 12:22:44.490872    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3806556c_d448_4e43_b1de_e0f254f1c36e.slice/crio-5de5814ccf146d5cc45fe01e56c2e268842b7fc5b25bea14fb2a4f78e67e5864 WatchSource:0}: Error finding container 5de5814ccf146d5cc45fe01e56c2e268842b7fc5b25bea14fb2a4f78e67e5864: Status 404 returned error can't find the container with id 5de5814ccf146d5cc45fe01e56c2e268842b7fc5b25bea14fb2a4f78e67e5864
May 26 12:22:44.492564 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.492537    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 12:22:44.576713 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:44.576675    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd" event={"ID":"3806556c-d448-4e43-b1de-e0f254f1c36e","Type":"ContainerStarted","Data":"5de5814ccf146d5cc45fe01e56c2e268842b7fc5b25bea14fb2a4f78e67e5864"}
May 26 12:22:45.582257 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:45.582221    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd" event={"ID":"3806556c-d448-4e43-b1de-e0f254f1c36e","Type":"ContainerStarted","Data":"5df7219722d173fdaf392ef7af3483d92d6e9d13a4f1bf7b162fa1fe29e54d1f"}
May 26 12:22:45.582647 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:45.582308    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:45.583896 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:45.583876    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5chfj_e73ee593-f6c2-4e31-9297-0c2c0b321c4a/dns/0.log"
May 26 12:22:45.598473 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:45.598436    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd" podStartSLOduration=1.59842226 podStartE2EDuration="1.59842226s" podCreationTimestamp="2026-05-26 12:22:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:22:45.597237303 +0000 UTC m=+3498.403075766" watchObservedRunningTime="2026-05-26 12:22:45.59842226 +0000 UTC m=+3498.404260761"
May 26 12:22:45.609080 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:45.609062    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5chfj_e73ee593-f6c2-4e31-9297-0c2c0b321c4a/kube-rbac-proxy/0.log"
May 26 12:22:45.743917 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:45.743885    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nqs65_9d3e68b4-5329-4f10-bf42-4f7e16a2795c/dns-node-resolver/0.log"
May 26 12:22:46.219408 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:46.219376    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8dl7t_1061f872-9b39-430c-9fe2-e1308e621e42/node-ca/0.log"
May 26 12:22:47.286944 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:47.286903    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rtrlh_3f9d7bb9-08d8-495f-9713-a6432e7f6a4f/serve-healthcheck-canary/0.log"
May 26 12:22:47.826388 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:47.826358    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rhx5w_fba481c9-7892-4902-aa77-9eb92d9e38b6/kube-rbac-proxy/0.log"
May 26 12:22:47.848043 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:47.848023    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rhx5w_fba481c9-7892-4902-aa77-9eb92d9e38b6/exporter/0.log"
May 26 12:22:47.869193 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:47.869164    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rhx5w_fba481c9-7892-4902-aa77-9eb92d9e38b6/extractor/0.log"
May 26 12:22:49.779750 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:49.779719    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7cd9b78f5d-skfch_55c5bd52-67a2-4519-a10d-caa351829609/manager/0.log"
May 26 12:22:49.800151 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:49.800109    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6b94ff949c-ckw25_79a85375-a60f-4f2e-b091-9f5b26139bd6/manager/0.log"
May 26 12:22:49.822210 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:49.822186    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-q622l_52ad2ceb-5698-46bd-aaee-6a8b0b5e96b8/server/0.log"
May 26 12:22:50.065792 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:50.065691    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-bwrpv_ca12c3cc-0728-498d-b182-d489f05cf877/manager/0.log"
May 26 12:22:50.111100 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:50.111069    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-9dgds_217ba037-a3c8-4c99-b59a-90a25b8b89aa/seaweedfs/0.log"
May 26 12:22:51.595623 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:51.595597    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zm7c5/perf-node-gather-daemonset-5srdd"
May 26 12:22:56.019397 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:56.019371    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rmwg_daf3e280-5478-4519-83a2-5e5a7af26aa4/kube-multus-additional-cni-plugins/0.log"
May 26 12:22:56.050424 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:56.050374    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rmwg_daf3e280-5478-4519-83a2-5e5a7af26aa4/egress-router-binary-copy/0.log"
May 26 12:22:56.093052 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:56.093020    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rmwg_daf3e280-5478-4519-83a2-5e5a7af26aa4/cni-plugins/0.log"
May 26 12:22:56.126966 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:56.126935    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rmwg_daf3e280-5478-4519-83a2-5e5a7af26aa4/bond-cni-plugin/0.log"
May 26 12:22:56.167094 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:56.167069    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rmwg_daf3e280-5478-4519-83a2-5e5a7af26aa4/routeoverride-cni/0.log"
May 26 12:22:56.212351 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:56.212264    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rmwg_daf3e280-5478-4519-83a2-5e5a7af26aa4/whereabouts-cni-bincopy/0.log"
May 26 12:22:56.251448 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:56.251415    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rmwg_daf3e280-5478-4519-83a2-5e5a7af26aa4/whereabouts-cni/0.log"
May 26 12:22:56.534055 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:56.534017    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fhpf8_3b46a615-dbda-42ba-ba5b-0ac8bb3076c8/kube-multus/0.log"
May 26 12:22:56.686411 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:56.686387    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vgwsb_a9201d70-b657-4adf-9d01-5363b9fb7ded/network-metrics-daemon/0.log"
May 26 12:22:56.720734 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:56.720707    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vgwsb_a9201d70-b657-4adf-9d01-5363b9fb7ded/kube-rbac-proxy/0.log"
May 26 12:22:57.573240 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:57.573214    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkfdl_a1d1b0b3-d529-499b-8126-e90cbc5a8ac4/ovn-controller/0.log"
May 26 12:22:57.607969 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:57.607942    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkfdl_a1d1b0b3-d529-499b-8126-e90cbc5a8ac4/ovn-acl-logging/0.log"
May 26 12:22:57.626031 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:57.625993    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkfdl_a1d1b0b3-d529-499b-8126-e90cbc5a8ac4/kube-rbac-proxy-node/0.log"
May 26 12:22:57.651062 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:57.651039    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkfdl_a1d1b0b3-d529-499b-8126-e90cbc5a8ac4/kube-rbac-proxy-ovn-metrics/0.log"
May 26 12:22:57.670870 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:57.670851    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkfdl_a1d1b0b3-d529-499b-8126-e90cbc5a8ac4/northd/0.log"
May 26 12:22:57.691887 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:57.691869    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkfdl_a1d1b0b3-d529-499b-8126-e90cbc5a8ac4/nbdb/0.log"
May 26 12:22:57.713810 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:57.713793    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkfdl_a1d1b0b3-d529-499b-8126-e90cbc5a8ac4/sbdb/0.log"
May 26 12:22:57.837790 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:57.837715    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkfdl_a1d1b0b3-d529-499b-8126-e90cbc5a8ac4/ovnkube-controller/0.log"
May 26 12:22:59.420705 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:22:59.420675    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-5pwxn_44da6752-3b21-42e6-942c-ab7fc44713dc/network-check-target-container/0.log"
May 26 12:23:00.426393 ip-10-0-135-71 kubenswrapper[2563]: I0526 12:23:00.426366    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-gv6w6_1c2dae63-45e0-481d-b9de-7eea78edbc6b/iptables-alerter/0.log"