Jul 23 13:40:50.248687 ip-10-0-138-125 systemd[1]: Starting Kubernetes Kubelet...
Jul 23 13:40:51.880051 ip-10-0-138-125 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 13:40:51.880051 ip-10-0-138-125 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 23 13:40:51.880051 ip-10-0-138-125 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 13:40:51.880051 ip-10-0-138-125 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 23 13:40:51.881370 ip-10-0-138-125 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 13:40:51.881498 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.881413    2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 23 13:40:51.887772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887752    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 13:40:51.887772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887771    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 13:40:51.887772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887774    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887778    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887781    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887784    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887787    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887790    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887792    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887795    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887797    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887810    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887813    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887816    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887818    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887821    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887824    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887827    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887829    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887832    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887834    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887837    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 13:40:51.887888 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887840    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887842    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887845    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887847    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887859    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887862    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887865    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887867    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887870    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887872    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887875    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887878    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887880    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887883    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887888    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887893    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887896    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887899    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887902    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 13:40:51.888384 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887905    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887907    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887910    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887917    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887920    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887923    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887926    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887929    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887932    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887935    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887938    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887940    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887943    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887946    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887948    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887951    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887959    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887962    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887965    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 13:40:51.888845 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887968    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887972    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887975    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887978    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887980    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887983    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887986    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887989    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887992    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887994    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.887997    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888000    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888002    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888005    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888007    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888012    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888016    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888020    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888023    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 13:40:51.889324 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888025    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888028    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888030    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888033    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888035    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888039    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888042    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888044    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888047    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888050    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888053    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888055    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888058    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888060    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888062    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888065    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888068    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888071    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888073    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888076    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 13:40:51.889772 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888887    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888896    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888899    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888902    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888905    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888908    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888911    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888913    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888916    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888919    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888922    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888925    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888928    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888930    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888933    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888936    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888938    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888941    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888945    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888948    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 13:40:51.890275 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888952    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888955    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888957    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888960    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888963    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888966    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888969    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888972    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888975    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888977    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888980    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888983    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888985    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888988    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888990    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888993    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888995    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.888998    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889000    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889003    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 13:40:51.890785 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889005    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889008    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889011    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889013    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889016    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889019    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889021    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889024    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889027    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889031    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889033    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889036    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889038    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889041    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889044    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889046    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889049    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889051    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889055    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889057    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 13:40:51.891277 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889060    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889062    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889065    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889068    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889071    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889073    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889076    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889079    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889081    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889085    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889089    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889092    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889094    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889097    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889099    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889102    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889104    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889107    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889110    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 13:40:51.891746 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889113    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889115    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889118    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889121    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889123    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889126    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889129    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889131    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889135    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889137    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889140    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889143    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889146    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889149    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889151    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889154    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889156    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889159    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889161    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889164    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 13:40:51.892210 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889545    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889549    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889552    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889554    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889557    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889560    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889562    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889565    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889567    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889569    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889573    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889575    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889578    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889581    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889583    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889586    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889589    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889592    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889594    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889597    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 13:40:51.892686 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889599    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889602    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889604    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889607    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889610    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889612    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889615    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889617    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889620    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889623    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889625    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889628    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889630    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889633    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889635    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889637    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889640    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889642    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889645    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889647    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 13:40:51.893174 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889650    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889652    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889655    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889658    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889660    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889663    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889665    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889668    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889670    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889674    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889679    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889683    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889686    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889689    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889692    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889695    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889698    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889700    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889703    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 13:40:51.893645 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889706    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889708    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889711    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889713    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889716    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889718    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889720    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889723    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889725    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889728    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889730    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889733    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889735    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889737    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889740    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889742    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889745    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889748    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889750    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889753    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 13:40:51.894226 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889755    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889758    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889761    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889765    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889768    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889771    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889773    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889776    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889781    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889784    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889787    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889789    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889792    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889794    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889797    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889799    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889802    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889804    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889807    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.889809    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 13:40:51.894711 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889904    2568 flags.go:64] FLAG: --address="0.0.0.0"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889912    2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889920    2568 flags.go:64] FLAG: --anonymous-auth="true"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889924    2568 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889934    2568 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889937    2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889941    2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889946    2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889949    2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889952    2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889956    2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889960    2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889965    2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889968    2568 flags.go:64] FLAG: --cgroup-root=""
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889971    2568 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889974    2568 flags.go:64] FLAG: --client-ca-file=""
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889977    2568 flags.go:64] FLAG: --cloud-provider="external"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889980    2568 flags.go:64] FLAG: --cluster-dns="[]"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889984    2568 flags.go:64] FLAG: --cluster-domain=""
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889987    2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889992    2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889995    2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.889999    2568 flags.go:64] FLAG: --container-log-max-files="5"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890002    2568 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 23 13:40:51.895199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890005    2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890008    2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890012    2568 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890015    2568 flags.go:64] FLAG: --contention-profiling="false"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890018    2568 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890021    2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890024    2568 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890027    2568 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890031    2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890034    2568 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890037    2568 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890040    2568 flags.go:64] FLAG: --enable-load-reader="false"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890043    2568 flags.go:64] FLAG: --enable-server="true"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890045    2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890051    2568 flags.go:64] FLAG: --event-burst="100"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890054    2568 flags.go:64] FLAG: --event-qps="50"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890057    2568 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890060    2568 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890063    2568 flags.go:64] FLAG: --eviction-hard=""
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890067    2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890071    2568 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890074    2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890077    2568 flags.go:64] FLAG: --eviction-soft=""
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890080    2568 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890083    2568 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 23 13:40:51.895764 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890086    2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890089    2568 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890092    2568 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890096    2568 flags.go:64] FLAG: --fail-swap-on="true"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890099    2568 flags.go:64] FLAG: --feature-gates=""
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890103    2568 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890106    2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890109    2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890112    2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890115    2568 flags.go:64] FLAG: --healthz-port="10248"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890119    2568 flags.go:64] FLAG: --help="false"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890122    2568 flags.go:64] FLAG: --hostname-override="ip-10-0-138-125.ec2.internal"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890125    2568 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890166    2568 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890172    2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890177    2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890182    2568 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890186    2568 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890189    2568 flags.go:64] FLAG: --image-service-endpoint=""
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890193    2568 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890197    2568 flags.go:64] FLAG: --kube-api-burst="100"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890200    2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890204    2568 flags.go:64] FLAG: --kube-api-qps="50"
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890207    2568 flags.go:64] FLAG: --kube-reserved=""
Jul 23 13:40:51.896428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890211    2568 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890214    2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890217    2568 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890220    2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890223    2568 flags.go:64] FLAG: --lock-file=""
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890226    2568 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890229    2568 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890232    2568 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890239    2568 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890242    2568 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890245    2568 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890248    2568 flags.go:64] FLAG: --logging-format="text"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890252    2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890255    2568 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890258    2568 flags.go:64] FLAG: --manifest-url=""
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890261    2568 flags.go:64] FLAG: --manifest-url-header=""
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890266    2568 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890269    2568 flags.go:64] FLAG: --max-open-files="1000000"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890274    2568 flags.go:64] FLAG: --max-pods="110"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890277    2568 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890280    2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890287    2568 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890291    2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890295    2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890298    2568 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 23 13:40:51.897079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890301    2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890309    2568 flags.go:64] FLAG: --node-status-max-images="50"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890312    2568 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890316    2568 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890319    2568 flags.go:64] FLAG: --pod-cidr=""
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890322    2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890328    2568 flags.go:64] FLAG: --pod-manifest-path=""
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890331    2568 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890334    2568 flags.go:64] FLAG: --pods-per-core="0"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890338    2568 flags.go:64] FLAG: --port="10250"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890340    2568 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890343    2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-088bbddf72f328060"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890346    2568 flags.go:64] FLAG: --qos-reserved=""
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890349    2568 flags.go:64] FLAG: --read-only-port="10255"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890353    2568 flags.go:64] FLAG: --register-node="true"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890356    2568 flags.go:64] FLAG: --register-with-taints=""
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890360    2568 flags.go:64] FLAG: --registry-burst="10"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890362    2568 flags.go:64] FLAG: --registry-qps="5"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890365    2568 flags.go:64] FLAG: --reserved-cpus=""
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890368    2568 flags.go:64] FLAG: --reserved-memory=""
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890372    2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890375    2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890378    2568 flags.go:64] FLAG: --rotate-certificates="false"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890381    2568 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890384    2568 flags.go:64] FLAG: --runonce="false"
Jul 23 13:40:51.897798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890386    2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890389    2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890392    2568 flags.go:64] FLAG: --seccomp-default="false"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890396    2568 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890402    2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890406    2568 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890409    2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890412    2568 flags.go:64] FLAG: --storage-driver-password="root"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890415    2568 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890418    2568 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890421    2568 flags.go:64] FLAG: --storage-driver-user="root"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890424    2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890427    2568 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890430    2568 flags.go:64] FLAG: --system-cgroups=""
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890433    2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890439    2568 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890442    2568 flags.go:64] FLAG: --tls-cert-file=""
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890445    2568 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890451    2568 flags.go:64] FLAG: --tls-min-version=""
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890454    2568 flags.go:64] FLAG: --tls-private-key-file=""
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890457    2568 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890460    2568 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890463    2568 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890466    2568 flags.go:64] FLAG: --v="2"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890475    2568 flags.go:64] FLAG: --version="false"
Jul 23 13:40:51.898742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890479    2568 flags.go:64] FLAG: --vmodule=""
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890483    2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890487    2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890653    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890658    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890661    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890664    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890667    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890670    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890673    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890677    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890682    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890685    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890695    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890699    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890702    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890704    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890708    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890712    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890715    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 13:40:51.899421 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890718    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890721    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890724    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890726    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890729    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890732    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890735    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890737    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890740    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890743    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890745    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890747    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890750    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890753    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890755    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890758    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890760    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890763    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890765    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890768    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 13:40:51.899962 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890770    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890774    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890780    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890784    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890787    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890790    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890799    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890802    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890805    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890807    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890810    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890813    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890815    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890818    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890820    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890823    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890825    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890828    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890830    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 13:40:51.900485 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890833    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890835    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890838    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890841    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890843    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890846    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890870    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890874    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890877    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890879    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890882    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890884    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890887    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890889    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890892    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890895    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890898    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890901    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890903    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890906    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 13:40:51.901047 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890915    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890919    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890923    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890926    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890928    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890931    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890933    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890936    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890938    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890941    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890943    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890946    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890948    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890951    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890953    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890955    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890958    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890960    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890963    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 13:40:51.901527 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890965    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 13:40:51.902002 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890967    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 13:40:51.902002 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890970    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 13:40:51.902002 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.890973    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 13:40:51.902002 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.890982    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 13:40:51.902002 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.900880    2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 23 13:40:51.902002 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.900899    2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 23 13:40:51.902156 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.902078    2568 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 23 13:40:51.902156 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.902086    2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902155    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902161    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902164    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902167    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902170    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902173    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902175    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902178    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902181    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902183    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902186    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902190    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902192    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902195    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902198    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902200    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902203    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902205    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902208    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 13:40:51.902208 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902211    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902214    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902216    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902219    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902223    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902225    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902228    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902230    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902233    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902235    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902238    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902241    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902243    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902246    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902249    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902253    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902258    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902261    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902265    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 13:40:51.902690 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902269    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902272    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902275    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902278    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902281    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902283    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902286    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902289    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902299    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902302    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902305    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902307    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902310    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902312    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902315    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902318    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902321    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902324    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902326    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902329    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 13:40:51.903158 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902331    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902335    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902337    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902340    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902343    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902345    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902348    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902351    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902353    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902356    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902359    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902361    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902365    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902368    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902372    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902375    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902377    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902380    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902383    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 13:40:51.903637 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902386    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902389    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902397    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902400    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902402    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902405    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902407    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902410    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902414    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902416    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902419    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902422    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902424    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902427    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902430    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902432    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902435    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902437    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902440    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902442    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 13:40:51.904106 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902445    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 13:40:51.904580 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902447    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 13:40:51.904580 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.902454    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 13:40:51.904580 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902577    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 13:40:51.904580 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902583    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 13:40:51.904580 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902588    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 13:40:51.904580 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902591    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 13:40:51.904580 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902594    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 13:40:51.904580 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902597    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 13:40:51.904580 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902599    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 13:40:51.904580 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902602    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 13:40:51.904580 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902605    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 13:40:51.904580 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902607    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 13:40:51.904580 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902610    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 13:40:51.904580 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902612    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 13:40:51.904580 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902619    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902622    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902624    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902627    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902629    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902632    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902635    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902638    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902640    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902643    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902646    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902648    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902651    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902654    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902656    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902659    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902661    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902664    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902667    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902669    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 13:40:51.905037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902673    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902676    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902679    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902682    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902684    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902687    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902690    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902692    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902695    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902698    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902701    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902704    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902706    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902710    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902712    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902715    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902717    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902720    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902722    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 13:40:51.905521 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902725    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902727    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902730    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902732    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902735    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902737    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902740    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902742    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902744    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902747    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902749    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902752    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902755    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902757    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902759    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902762    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902764    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902767    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902769    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902772    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 13:40:51.906026 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902774    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902777    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902779    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902782    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902785    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902788    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902790    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902793    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902796    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902798    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902801    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902803    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902805    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902808    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902810    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902813    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902815    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902818    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902820    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902822    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 13:40:51.906507 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902825    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 13:40:51.906995 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902827    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 13:40:51.906995 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902830    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 13:40:51.906995 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902832    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 13:40:51.906995 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902835    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 13:40:51.906995 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902837    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 13:40:51.906995 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902840    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 13:40:51.906995 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:51.902844    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 13:40:51.906995 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.902863    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 13:40:51.906995 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.902986    2568 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 23 13:40:51.906995 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.905417    2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 23 13:40:51.906995 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.906965    2568 server.go:1019] "Starting client certificate rotation"
Jul 23 13:40:51.907270 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.907064    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 13:40:51.907270 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.907093    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 13:40:51.909890 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.909876    2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 13:40:51.911220 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.911198    2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 13:40:51.929654 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.929630    2568 log.go:25] "Validated CRI v1 runtime API"
Jul 23 13:40:51.932259 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.932240    2568 log.go:25] "Validated CRI v1 image API"
Jul 23 13:40:51.933079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.933064    2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 23 13:40:51.936454 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.936409    2568 fs.go:135] Filesystem UUIDs: map[41dfd76c-67be-4ebe-9f05-9ee53b4dc9c1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a1f2940f-2104-4549-b09f-965357ebff36:/dev/nvme0n1p3]
Jul 23 13:40:51.936518 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.936453    2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 23 13:40:51.942062 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.941947    2568 manager.go:216] Machine: {Timestamp:2026-07-23 13:40:51.941422817 +0000 UTC m=+0.402513850 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3129420 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec236531ca39b0ae9369d1fb6e7a267a SystemUUID:ec236531-ca39-b0ae-9369-d1fb6e7a267a BootID:351a41af-a05f-4c57-b6c5-58a5800227d4 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:be:04:33:ef:99 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:be:04:33:ef:99 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:ec:6d:d6:3e:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 23 13:40:51.942062 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.942056    2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 23 13:40:51.942218 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.942198    2568 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 23 13:40:51.942665 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.942617    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 13:40:51.946082 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.945879    2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 23 13:40:51.946241 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.946087    2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-125.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 23 13:40:51.947655 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.947644    2568 topology_manager.go:138] "Creating topology manager with none policy"
Jul 23 13:40:51.947707 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.947657    2568 container_manager_linux.go:309] "Creating device plugin manager"
Jul 23 13:40:51.947707 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.947664    2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 13:40:51.948611 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.948601    2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 13:40:51.950177 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.950167    2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 23 13:40:51.953919 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.953908    2568 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 13:40:51.954056 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.954046    2568 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 23 13:40:51.954095 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.954076    2568 kubelet.go:486] "Attempting to sync node with API server"
Jul 23 13:40:51.954095 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.954086    2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 23 13:40:51.954166 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.954099    2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 23 13:40:51.954166 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.954108    2568 kubelet.go:398] "Adding apiserver pod source"
Jul 23 13:40:51.954166 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.954116    2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 23 13:40:51.954489 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.954474    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 13:40:51.954543 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.954495    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 13:40:51.957302 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.957286    2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 23 13:40:51.959696 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.959681    2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 23 13:40:51.959772 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.959712    2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 23 13:40:51.961261 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.961246    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 23 13:40:51.961308 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.961273    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 23 13:40:51.961308 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.961282    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 23 13:40:51.961308 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.961288    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 23 13:40:51.961308 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.961297    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 23 13:40:51.961308 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.961303    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 23 13:40:51.961446 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.961311    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 23 13:40:51.961446 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.961318    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 23 13:40:51.961446 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.961324    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 23 13:40:51.961446 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.961330    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 23 13:40:51.961446 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.961339    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 23 13:40:51.961446 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.961346    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 23 13:40:51.962356 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.962342    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 23 13:40:51.962417 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.962359    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 23 13:40:51.964206 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.964190    2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-125.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 23 13:40:51.964554 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:51.964534    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 23 13:40:51.964591 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:51.964562    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 23 13:40:51.965467 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.965456    2568 server.go:1268] "Started kubelet"
Jul 23 13:40:51.965585 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.965531    2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 23 13:40:51.965626 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.965610    2568 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 23 13:40:51.966343 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.966308    2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 23 13:40:51.966383 ip-10-0-138-125 systemd[1]: Started Kubernetes Kubelet.
Jul 23 13:40:51.967482 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.967396    2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 23 13:40:51.968745 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.968724    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nzljk"
Jul 23 13:40:51.968921 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.968903    2568 server.go:310] "Adding debug handlers to kubelet server"
Jul 23 13:40:51.969689 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:51.969651    2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 23 13:40:51.974909 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.974893    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nzljk"
Jul 23 13:40:51.975132 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.975113    2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 23 13:40:51.975132 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.975122    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 23 13:40:51.975246 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:51.975227    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-125.ec2.internal\" not found"
Jul 23 13:40:51.975988 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.975970    2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 23 13:40:51.976087 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.975990    2568 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 23 13:40:51.976087 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.976002    2568 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 23 13:40:51.976158 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.976112    2568 reconstruct.go:97] "Volume reconstruction finished"
Jul 23 13:40:51.976158 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.976123    2568 reconciler.go:29] "Reconciler: start to sync state"
Jul 23 13:40:51.976664 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.976645    2568 factory.go:153] Registering CRI-O factory
Jul 23 13:40:51.976664 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.976665    2568 factory.go:223] Registration of the crio container factory successfully
Jul 23 13:40:51.976765 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.976720    2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 23 13:40:51.976765 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.976730    2568 factory.go:55] Registering systemd factory
Jul 23 13:40:51.976765 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.976739    2568 factory.go:223] Registration of the systemd container factory successfully
Jul 23 13:40:51.976890 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.976772    2568 factory.go:103] Registering Raw factory
Jul 23 13:40:51.976890 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.976787    2568 manager.go:1196] Started watching for new ooms in manager
Jul 23 13:40:51.977143 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.977133    2568 manager.go:349] Starting recovery of all containers
Jul 23 13:40:51.982640 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:51.981427    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-125.ec2.internal.18c4eeec85866545  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-125.ec2.internal,UID:ip-10-0-138-125.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-125.ec2.internal,},FirstTimestamp:2026-07-23 13:40:51.965429061 +0000 UTC m=+0.426520095,LastTimestamp:2026-07-23 13:40:51.965429061 +0000 UTC m=+0.426520095,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-125.ec2.internal,}"
Jul 23 13:40:51.985083 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:51.985054    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 23 13:40:51.985758 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:51.985518    2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 23 13:40:51.989514 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.989500    2568 manager.go:354] Recovery completed
Jul 23 13:40:51.991059 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:51.991041    2568 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 23 13:40:51.994012 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.994001    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 13:40:51.996114 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.996097    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 13:40:51.996206 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.996129    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 13:40:51.996206 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.996141    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasSufficientPID"
Jul 23 13:40:51.996705 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.996692    2568 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 23 13:40:51.996705 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.996704    2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 23 13:40:51.996785 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.996721    2568 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 13:40:51.996785 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.996747    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 13:40:51.997706 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.997692    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 13:40:51.997779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.997718    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 13:40:51.997779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.997727    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasSufficientPID"
Jul 23 13:40:51.998140 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.998129    2568 policy_none.go:49] "None policy: Start"
Jul 23 13:40:51.998182 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.998145    2568 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 23 13:40:51.998182 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.998155    2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 23 13:40:51.999444 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:51.999433    2568 policy_none.go:47] "Start"
Jul 23 13:40:52.035729 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.035531    2568 manager.go:337] "Starting Device Plugin manager"
Jul 23 13:40:52.035836 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.035750    2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 23 13:40:52.035836 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.035762    2568 server.go:87] "Starting device plugin registration server"
Jul 23 13:40:52.036017 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.036006    2568 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 23 13:40:52.036051 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.036018    2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 23 13:40:52.036148 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.036126    2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 23 13:40:52.036264 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.036215    2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 23 13:40:52.036264 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.036224    2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 23 13:40:52.036794 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.036767    2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 23 13:40:52.036909 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.036822    2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-125.ec2.internal\" not found"
Jul 23 13:40:52.136395 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.136351    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 13:40:52.137705 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.137393    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 13:40:52.137705 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.137425    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 13:40:52.137705 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.137435    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasSufficientPID"
Jul 23 13:40:52.137705 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.137464    2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.143802 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.143771    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 23 13:40:52.144193 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.144176    2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.144296 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.144199    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-125.ec2.internal\": node \"ip-10-0-138-125.ec2.internal\" not found"
Jul 23 13:40:52.144991 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.144974    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 23 13:40:52.145036 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.145014    2568 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 23 13:40:52.145068 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.145047    2568 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 23 13:40:52.145099 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.145090    2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 23 13:40:52.146937 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.146916    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 13:40:52.191600 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.191574    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-125.ec2.internal\" not found"
Jul 23 13:40:52.245888 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.245819    2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-125.ec2.internal"]
Jul 23 13:40:52.246059 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.245931    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 13:40:52.247614 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.247598    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 13:40:52.247675 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.247628    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 13:40:52.247675 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.247638    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasSufficientPID"
Jul 23 13:40:52.248974 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.248961    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 13:40:52.249127 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.249113    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.249163 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.249145    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 13:40:52.249702 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.249688    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 13:40:52.249702 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.249698    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 13:40:52.249788 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.249719    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 13:40:52.249788 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.249731    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasSufficientPID"
Jul 23 13:40:52.249845 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.249788    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 13:40:52.249845 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.249807    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasSufficientPID"
Jul 23 13:40:52.251100 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.251087    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.251154 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.251118    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 13:40:52.251822 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.251804    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 13:40:52.251935 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.251834    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 13:40:52.251935 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.251863    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeHasSufficientPID"
Jul 23 13:40:52.278141 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.278114    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3de01a743a50264260f9a41b54c7e4a3-config\") pod \"kube-apiserver-proxy-ip-10-0-138-125.ec2.internal\" (UID: \"3de01a743a50264260f9a41b54c7e4a3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.278226 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.278146    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/08b7d7feb9d6abfd35e59891c87373ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal\" (UID: \"08b7d7feb9d6abfd35e59891c87373ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.278226 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.278163    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08b7d7feb9d6abfd35e59891c87373ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal\" (UID: \"08b7d7feb9d6abfd35e59891c87373ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.280707 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.280688    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-125.ec2.internal\" not found" node="ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.285282 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.285266    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-125.ec2.internal\" not found" node="ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.291946 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.291928    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-125.ec2.internal\" not found"
Jul 23 13:40:52.378333 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.378262    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3de01a743a50264260f9a41b54c7e4a3-config\") pod \"kube-apiserver-proxy-ip-10-0-138-125.ec2.internal\" (UID: \"3de01a743a50264260f9a41b54c7e4a3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.378522 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.378369    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/08b7d7feb9d6abfd35e59891c87373ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal\" (UID: \"08b7d7feb9d6abfd35e59891c87373ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.378522 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.378392    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08b7d7feb9d6abfd35e59891c87373ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal\" (UID: \"08b7d7feb9d6abfd35e59891c87373ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.378522 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.378299    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3de01a743a50264260f9a41b54c7e4a3-config\") pod \"kube-apiserver-proxy-ip-10-0-138-125.ec2.internal\" (UID: \"3de01a743a50264260f9a41b54c7e4a3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.378522 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.378445    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/08b7d7feb9d6abfd35e59891c87373ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal\" (UID: \"08b7d7feb9d6abfd35e59891c87373ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.378522 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.378473    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08b7d7feb9d6abfd35e59891c87373ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal\" (UID: \"08b7d7feb9d6abfd35e59891c87373ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.392150 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.392121    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-125.ec2.internal\" not found"
Jul 23 13:40:52.492987 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.492952    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-125.ec2.internal\" not found"
Jul 23 13:40:52.585551 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.585476    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.588055 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.588039    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.593994 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.593972    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-125.ec2.internal\" not found"
Jul 23 13:40:52.694487 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.694446    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-125.ec2.internal\" not found"
Jul 23 13:40:52.795021 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.794982    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-125.ec2.internal\" not found"
Jul 23 13:40:52.895539 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.895466    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-125.ec2.internal\" not found"
Jul 23 13:40:52.906640 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.906616    2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 23 13:40:52.906777 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.906760    2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 13:40:52.927970 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.927947    2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 13:40:52.953977 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.953957    2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 13:40:52.954907 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.954892    2568 apiserver.go:52] "Watching apiserver"
Jul 23 13:40:52.964949 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.964928    2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 23 13:40:52.966631 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.966600    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-44tr7","openshift-multus/network-metrics-daemon-88prr","openshift-ovn-kubernetes/ovnkube-node-bswct","kube-system/global-pull-secret-syncer-fvtk6","kube-system/konnectivity-agent-xvlj5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2","openshift-image-registry/node-ca-ddfgb","openshift-multus/multus-qt824","openshift-network-diagnostics/network-check-target-6fh9p","openshift-network-operator/iptables-alerter-j625q","openshift-cluster-node-tuning-operator/tuned-r5649","openshift-dns/node-resolver-9tk9p"]
Jul 23 13:40:52.968993 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.968969    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:52.971160 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.971141    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:40:52.971254 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.971198    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:40:52.971568 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.971552    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 23 13:40:52.971655 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.971568    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 23 13:40:52.971655 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.971576    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 23 13:40:52.971655 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.971558    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 23 13:40:52.971874 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.971711    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 23 13:40:52.971934 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.971920    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8qxnw\""
Jul 23 13:40:52.972688 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.972673    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.974250 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.974232    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:40:52.974322 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.974298    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:40:52.974923 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.974910    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f6k7q\""
Jul 23 13:40:52.975592 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.975581    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 23 13:40:52.975629 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.975617    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 23 13:40:52.975702 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.975620    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-125.ec2.internal"
Jul 23 13:40:52.975926 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.975910    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xvlj5"
Jul 23 13:40:52.976238 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.976223    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 23 13:40:52.976441 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.976425    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 23 13:40:52.976482 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.976473    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 23 13:40:52.976524 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.976498    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 23 13:40:52.976524 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.976514    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 23 13:40:52.977427 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.977411    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:52.979137 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.979121    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ddfgb"
Jul 23 13:40:52.979316 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.979277    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 13:35:51 +0000 UTC" deadline="2028-03-14 21:09:53.023755657 +0000 UTC"
Jul 23 13:40:52.979372 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.979316    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14407h29m0.04444323s"
Jul 23 13:40:52.980566 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.980551    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 23 13:40:52.980566 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.980552    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qt824"
Jul 23 13:40:52.981631 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.981612    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/092f6d96-3452-48f0-a66e-0866fde1c1b6-cnibin\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:52.981696 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.981643    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-systemd-units\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.981696 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.981667    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-run-systemd\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.981696 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.981691    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-log-socket\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.981817 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.981720    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/932faad9-0c93-47e5-a018-b2b4e79aba10-agent-certs\") pod \"konnectivity-agent-xvlj5\" (UID: \"932faad9-0c93-47e5-a018-b2b4e79aba10\") " pod="kube-system/konnectivity-agent-xvlj5"
Jul 23 13:40:52.981817 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.981743    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-socket-dir\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:52.981817 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.981764    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-kubelet\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.981973 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.981812    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-run-openvswitch\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.981973 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.981879    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/374f4643-8632-47b5-ae94-e9be6e6d6c4c-env-overrides\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.981973 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.981905    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-registration-dir\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:52.981973 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.981925    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-device-dir\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:52.981973 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.981961    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42qmh\" (UniqueName: \"kubernetes.io/projected/fc8ca143-ff44-4040-ba3c-fd70df8f6439-kube-api-access-42qmh\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:52.982182 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982030    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/092f6d96-3452-48f0-a66e-0866fde1c1b6-system-cni-dir\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:52.982182 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982055    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:40:52.982182 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982056    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.982182 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982073    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-cni-bin\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.982182 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:52.982098    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:40:52.982182 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982111    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:52.982182 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982155    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/092f6d96-3452-48f0-a66e-0866fde1c1b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:52.982182 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982179    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-slash\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.982474 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982207    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-run-ovn\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.982474 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982234    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/374f4643-8632-47b5-ae94-e9be6e6d6c4c-ovn-node-metrics-cert\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.982474 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982258    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-kubelet-config\") pod \"global-pull-secret-syncer-fvtk6\" (UID: \"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1\") " pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:40:52.982474 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982288    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/932faad9-0c93-47e5-a018-b2b4e79aba10-konnectivity-ca\") pod \"konnectivity-agent-xvlj5\" (UID: \"932faad9-0c93-47e5-a018-b2b4e79aba10\") " pod="kube-system/konnectivity-agent-xvlj5"
Jul 23 13:40:52.982474 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982317    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/092f6d96-3452-48f0-a66e-0866fde1c1b6-os-release\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:52.982474 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982339    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/092f6d96-3452-48f0-a66e-0866fde1c1b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:52.982474 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982363    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/092f6d96-3452-48f0-a66e-0866fde1c1b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:52.982474 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982382    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-var-lib-openvswitch\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.982474 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982396    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/374f4643-8632-47b5-ae94-e9be6e6d6c4c-ovnkube-script-lib\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.982474 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982414    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret\") pod \"global-pull-secret-syncer-fvtk6\" (UID: \"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1\") " pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:40:52.982474 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982435    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-sys-fs\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:52.982821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982485    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-etc-openvswitch\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.982821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982523    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 23 13:40:52.982821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982544    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs4vj\" (UniqueName: \"kubernetes.io/projected/374f4643-8632-47b5-ae94-e9be6e6d6c4c-kube-api-access-fs4vj\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.982821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982571    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-dbus\") pod \"global-pull-secret-syncer-fvtk6\" (UID: \"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1\") " pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:40:52.982821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982595    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-etc-selinux\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:52.982821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982618    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/092f6d96-3452-48f0-a66e-0866fde1c1b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:52.982821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982642    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.982821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982674    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-cni-netd\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.982821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982692    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9tsb\" (UniqueName: \"kubernetes.io/projected/092f6d96-3452-48f0-a66e-0866fde1c1b6-kube-api-access-n9tsb\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:52.982821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982706    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs\") pod \"network-metrics-daemon-88prr\" (UID: \"2a8b633a-9217-4779-89f3-893586369e2e\") " pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:40:52.982821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982726    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddrpp\" (UniqueName: \"kubernetes.io/projected/2a8b633a-9217-4779-89f3-893586369e2e-kube-api-access-ddrpp\") pod \"network-metrics-daemon-88prr\" (UID: \"2a8b633a-9217-4779-89f3-893586369e2e\") " pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:40:52.982821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982751    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-run-netns\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.982821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982804    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-node-log\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.983233 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.982829    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/374f4643-8632-47b5-ae94-e9be6e6d6c4c-ovnkube-config\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:52.983233 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.983054    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 23 13:40:52.983233 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.983091    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 23 13:40:52.983568 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.983537    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-btdbd\""
Jul 23 13:40:52.983669 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.983572    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 23 13:40:52.983669 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.983591    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m76cg\""
Jul 23 13:40:52.983669 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.983602    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 23 13:40:52.983669 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.983572    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 23 13:40:52.983669 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.983649    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 23 13:40:52.983931 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.983789    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j625q"
Jul 23 13:40:52.984224 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.984208    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 23 13:40:52.984298 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.984213    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cns9h\""
Jul 23 13:40:52.985388 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.985374    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:52.986521 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.986506    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 23 13:40:52.986604 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.986526    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-46kdf\""
Jul 23 13:40:52.986604 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.986511    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 23 13:40:52.986709 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.986530    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bkvmc\""
Jul 23 13:40:52.986709 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.986509    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 23 13:40:52.986939 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.986922    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9tk9p"
Jul 23 13:40:52.987440 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.987418    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 23 13:40:52.988468 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.988451    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 23 13:40:52.988560 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.988545    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gws4w\""
Jul 23 13:40:52.989652 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.989639    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 23 13:40:52.990137 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.990125    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xvl4f\""
Jul 23 13:40:52.991322 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.991310    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 23 13:40:52.993699 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.993685    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-125.ec2.internal"]
Jul 23 13:40:52.994603 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.994591    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 13:40:52.994651 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:52.994642    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal"
Jul 23 13:40:53.004730 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.004708    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 13:40:53.011490 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.011474    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 13:40:53.011617 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.011602    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal"]
Jul 23 13:40:53.057098 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.057073    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jdqr5"
Jul 23 13:40:53.070320 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.070298    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jdqr5"
Jul 23 13:40:53.076962 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.076942    2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 23 13:40:53.083313 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083292    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-kubelet\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.083397 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083317    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-run-openvswitch\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.083397 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083333    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/374f4643-8632-47b5-ae94-e9be6e6d6c4c-env-overrides\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.083397 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083350    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-registration-dir\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:53.083397 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083366    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-lib-modules\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.083397 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083381    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0489f3dd-bc92-4289-a74d-ceb26a4ee20e-host\") pod \"node-ca-ddfgb\" (UID: \"0489f3dd-bc92-4289-a74d-ceb26a4ee20e\") " pod="openshift-image-registry/node-ca-ddfgb"
Jul 23 13:40:53.083546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083415    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-kubelet\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.083546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083439    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-registration-dir\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:53.083546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083475    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-var-lib-cni-bin\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.083546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083498    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-run-openvswitch\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.083546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083503    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/092f6d96-3452-48f0-a66e-0866fde1c1b6-system-cni-dir\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.083546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083533    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/092f6d96-3452-48f0-a66e-0866fde1c1b6-system-cni-dir\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.083546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083542    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.083746 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083561    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:53.083746 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083579    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-system-cni-dir\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.083746 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083593    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-os-release\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.083746 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083592    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.083746 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083607    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-slash\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.083746 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083635    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-slash\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.083746 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083651    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-run-ovn\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.083746 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083667    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/374f4643-8632-47b5-ae94-e9be6e6d6c4c-ovn-node-metrics-cert\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.083746 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083682    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-kubelet-config\") pod \"global-pull-secret-syncer-fvtk6\" (UID: \"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1\") " pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:40:53.083746 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083696    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-run\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.083746 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083706    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-run-ovn\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.083746 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083717    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-cnibin\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.083746 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083731    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-kubelet-config\") pod \"global-pull-secret-syncer-fvtk6\" (UID: \"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1\") " pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:40:53.083746 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083742    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/092f6d96-3452-48f0-a66e-0866fde1c1b6-os-release\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.083746 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083747    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:53.084359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083767    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/092f6d96-3452-48f0-a66e-0866fde1c1b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.084359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083782    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/092f6d96-3452-48f0-a66e-0866fde1c1b6-os-release\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.084359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083793    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/374f4643-8632-47b5-ae94-e9be6e6d6c4c-ovnkube-script-lib\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.084359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083809    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-multus-daemon-config\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.084359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083830    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-etc-openvswitch\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.084359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083872    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs4vj\" (UniqueName: \"kubernetes.io/projected/374f4643-8632-47b5-ae94-e9be6e6d6c4c-kube-api-access-fs4vj\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.084359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083895    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-etc-selinux\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:53.084359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083920    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbfkl\" (UniqueName: \"kubernetes.io/projected/640cd623-c590-466c-8c27-be9913ba96c5-kube-api-access-nbfkl\") pod \"iptables-alerter-j625q\" (UID: \"640cd623-c590-466c-8c27-be9913ba96c5\") " pod="openshift-network-operator/iptables-alerter-j625q"
Jul 23 13:40:53.084359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083918    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-etc-openvswitch\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.084359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.083939    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-sysctl-d\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.084359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084011    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-etc-selinux\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:53.084359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084034    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.084359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084050    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-multus-conf-dir\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.084359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084065    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-cni-netd\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.084359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084079    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9tsb\" (UniqueName: \"kubernetes.io/projected/092f6d96-3452-48f0-a66e-0866fde1c1b6-kube-api-access-n9tsb\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.084359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084094    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs\") pod \"network-metrics-daemon-88prr\" (UID: \"2a8b633a-9217-4779-89f3-893586369e2e\") " pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084105    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084109    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-run-netns\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084132    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-run-netns\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084133    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-node-log\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084149    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-node-log\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084170    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/374f4643-8632-47b5-ae94-e9be6e6d6c4c-ovnkube-config\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084197    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-multus-socket-dir-parent\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084232    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-var-lib-kubelet\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084303    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-cni-netd\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084393    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/092f6d96-3452-48f0-a66e-0866fde1c1b6-cnibin\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:53.084478    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084500    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-systemd-units\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084523    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/092f6d96-3452-48f0-a66e-0866fde1c1b6-cnibin\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084535    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-run-systemd\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084565    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-log-socket\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084587    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-systemd-units\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084593    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/932faad9-0c93-47e5-a018-b2b4e79aba10-agent-certs\") pod \"konnectivity-agent-xvlj5\" (UID: \"932faad9-0c93-47e5-a018-b2b4e79aba10\") " pod="kube-system/konnectivity-agent-xvlj5"
Jul 23 13:40:53.085090 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084633    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-run-systemd\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084648    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-socket-dir\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:53.084676    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs podName:2a8b633a-9217-4779-89f3-893586369e2e nodeName:}" failed. No retries permitted until 2026-07-23 13:40:53.584650076 +0000 UTC m=+2.045741118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs") pod "network-metrics-daemon-88prr" (UID: "2a8b633a-9217-4779-89f3-893586369e2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084707    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-log-socket\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084693    2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084745    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-device-dir\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084752    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/374f4643-8632-47b5-ae94-e9be6e6d6c4c-env-overrides\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084766    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-socket-dir\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084773    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42qmh\" (UniqueName: \"kubernetes.io/projected/fc8ca143-ff44-4040-ba3c-fd70df8f6439-kube-api-access-42qmh\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084913    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/374f4643-8632-47b5-ae94-e9be6e6d6c4c-ovnkube-config\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084937    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/374f4643-8632-47b5-ae94-e9be6e6d6c4c-ovnkube-script-lib\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084973    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-device-dir\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084972    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c329870-9269-4992-9b11-d3dac8faba35-tmp\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.084987    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/092f6d96-3452-48f0-a66e-0866fde1c1b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085009    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5-tmp-dir\") pod \"node-resolver-9tk9p\" (UID: \"12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5\") " pod="openshift-dns/node-resolver-9tk9p"
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085030    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-multus-cni-dir\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085072    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-run-netns\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.085906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085090    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-hostroot\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.086546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085107    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m75fg\" (UniqueName: \"kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg\") pod \"network-check-target-6fh9p\" (UID: \"cf03b6c6-d0e9-4040-91d7-c33ddbe30d03\") " pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:40:53.086546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085124    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-cni-bin\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.086546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085164    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-etc-kubernetes\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.086546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085192    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-host-cni-bin\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.086546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085192    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/092f6d96-3452-48f0-a66e-0866fde1c1b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.086546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085331    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/932faad9-0c93-47e5-a018-b2b4e79aba10-konnectivity-ca\") pod \"konnectivity-agent-xvlj5\" (UID: \"932faad9-0c93-47e5-a018-b2b4e79aba10\") " pod="kube-system/konnectivity-agent-xvlj5"
Jul 23 13:40:53.086546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085362    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/640cd623-c590-466c-8c27-be9913ba96c5-iptables-alerter-script\") pod \"iptables-alerter-j625q\" (UID: \"640cd623-c590-466c-8c27-be9913ba96c5\") " pod="openshift-network-operator/iptables-alerter-j625q"
Jul 23 13:40:53.086546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085387    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/640cd623-c590-466c-8c27-be9913ba96c5-host-slash\") pod \"iptables-alerter-j625q\" (UID: \"640cd623-c590-466c-8c27-be9913ba96c5\") " pod="openshift-network-operator/iptables-alerter-j625q"
Jul 23 13:40:53.086546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085408    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-sysconfig\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.086546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085429    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-kubernetes\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.086546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085449    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c329870-9269-4992-9b11-d3dac8faba35-etc-tuned\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.086546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085468    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7696\" (UniqueName: \"kubernetes.io/projected/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-kube-api-access-n7696\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.086546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085491    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/092f6d96-3452-48f0-a66e-0866fde1c1b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.086546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085512    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-var-lib-openvswitch\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.086546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085531    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret\") pod \"global-pull-secret-syncer-fvtk6\" (UID: \"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1\") " pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:40:53.086546 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085551    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-sys-fs\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:53.087034 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085570    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx9nj\" (UniqueName: \"kubernetes.io/projected/7c329870-9269-4992-9b11-d3dac8faba35-kube-api-access-vx9nj\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.087034 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085590    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-669cc\" (UniqueName: \"kubernetes.io/projected/0489f3dd-bc92-4289-a74d-ceb26a4ee20e-kube-api-access-669cc\") pod \"node-ca-ddfgb\" (UID: \"0489f3dd-bc92-4289-a74d-ceb26a4ee20e\") " pod="openshift-image-registry/node-ca-ddfgb"
Jul 23 13:40:53.087034 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085596    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/374f4643-8632-47b5-ae94-e9be6e6d6c4c-var-lib-openvswitch\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.087034 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085611    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-dbus\") pod \"global-pull-secret-syncer-fvtk6\" (UID: \"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1\") " pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:40:53.087034 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085630    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-systemd\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.087034 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085652    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-run-multus-certs\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.087034 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085673    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/092f6d96-3452-48f0-a66e-0866fde1c1b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.087034 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085696    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-modprobe-d\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.087034 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085720    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-sysctl-conf\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.087034 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085739    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/092f6d96-3452-48f0-a66e-0866fde1c1b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.087034 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085821    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc8ca143-ff44-4040-ba3c-fd70df8f6439-sys-fs\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:53.087034 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085832    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/932faad9-0c93-47e5-a018-b2b4e79aba10-konnectivity-ca\") pod \"konnectivity-agent-xvlj5\" (UID: \"932faad9-0c93-47e5-a018-b2b4e79aba10\") " pod="kube-system/konnectivity-agent-xvlj5"
Jul 23 13:40:53.087034 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085904    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-dbus\") pod \"global-pull-secret-syncer-fvtk6\" (UID: \"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1\") " pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:40:53.087034 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:53.085917    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 13:40:53.087034 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085943    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-sys\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.087034 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:53.085963    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret podName:d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1 nodeName:}" failed. No retries permitted until 2026-07-23 13:40:53.585946929 +0000 UTC m=+2.047037991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret") pod "global-pull-secret-syncer-fvtk6" (UID: "d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1") : object "kube-system"/"original-pull-secret" not registered
Jul 23 13:40:53.087524 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.085995    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0489f3dd-bc92-4289-a74d-ceb26a4ee20e-serviceca\") pod \"node-ca-ddfgb\" (UID: \"0489f3dd-bc92-4289-a74d-ceb26a4ee20e\") " pod="openshift-image-registry/node-ca-ddfgb"
Jul 23 13:40:53.087524 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.086020    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-var-lib-cni-multus\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.087524 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.086042    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/092f6d96-3452-48f0-a66e-0866fde1c1b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.087524 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.086051    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddrpp\" (UniqueName: \"kubernetes.io/projected/2a8b633a-9217-4779-89f3-893586369e2e-kube-api-access-ddrpp\") pod \"network-metrics-daemon-88prr\" (UID: \"2a8b633a-9217-4779-89f3-893586369e2e\") " pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:40:53.087524 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.086077    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-host\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.087524 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.086095    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjfzw\" (UniqueName: \"kubernetes.io/projected/12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5-kube-api-access-sjfzw\") pod \"node-resolver-9tk9p\" (UID: \"12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5\") " pod="openshift-dns/node-resolver-9tk9p"
Jul 23 13:40:53.087524 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.086118    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-cni-binary-copy\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.087524 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.086146    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-var-lib-kubelet\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.087524 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.086161    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5-hosts-file\") pod \"node-resolver-9tk9p\" (UID: \"12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5\") " pod="openshift-dns/node-resolver-9tk9p"
Jul 23 13:40:53.087524 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.086176    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-run-k8s-cni-cncf-io\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.087524 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.086216    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/092f6d96-3452-48f0-a66e-0866fde1c1b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.087985 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.087967    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/374f4643-8632-47b5-ae94-e9be6e6d6c4c-ovn-node-metrics-cert\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.088120 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.088103    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/932faad9-0c93-47e5-a018-b2b4e79aba10-agent-certs\") pod \"konnectivity-agent-xvlj5\" (UID: \"932faad9-0c93-47e5-a018-b2b4e79aba10\") " pod="kube-system/konnectivity-agent-xvlj5"
Jul 23 13:40:53.101750 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.101726    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddrpp\" (UniqueName: \"kubernetes.io/projected/2a8b633a-9217-4779-89f3-893586369e2e-kube-api-access-ddrpp\") pod \"network-metrics-daemon-88prr\" (UID: \"2a8b633a-9217-4779-89f3-893586369e2e\") " pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:40:53.103089 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.103067    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9tsb\" (UniqueName: \"kubernetes.io/projected/092f6d96-3452-48f0-a66e-0866fde1c1b6-kube-api-access-n9tsb\") pod \"multus-additional-cni-plugins-44tr7\" (UID: \"092f6d96-3452-48f0-a66e-0866fde1c1b6\") " pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.103173 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.103148    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs4vj\" (UniqueName: \"kubernetes.io/projected/374f4643-8632-47b5-ae94-e9be6e6d6c4c-kube-api-access-fs4vj\") pod \"ovnkube-node-bswct\" (UID: \"374f4643-8632-47b5-ae94-e9be6e6d6c4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.103173 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.103148    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42qmh\" (UniqueName: \"kubernetes.io/projected/fc8ca143-ff44-4040-ba3c-fd70df8f6439-kube-api-access-42qmh\") pod \"aws-ebs-csi-driver-node-xssf2\" (UID: \"fc8ca143-ff44-4040-ba3c-fd70df8f6439\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:53.186908 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.186810    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-var-lib-kubelet\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.186908 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.186842    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5-hosts-file\") pod \"node-resolver-9tk9p\" (UID: \"12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5\") " pod="openshift-dns/node-resolver-9tk9p"
Jul 23 13:40:53.186908 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.186877    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-run-k8s-cni-cncf-io\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.186908 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.186903    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-lib-modules\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.187200 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.186924    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0489f3dd-bc92-4289-a74d-ceb26a4ee20e-host\") pod \"node-ca-ddfgb\" (UID: \"0489f3dd-bc92-4289-a74d-ceb26a4ee20e\") " pod="openshift-image-registry/node-ca-ddfgb"
Jul 23 13:40:53.187200 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.186940    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-var-lib-kubelet\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.187200 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.186962    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5-hosts-file\") pod \"node-resolver-9tk9p\" (UID: \"12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5\") " pod="openshift-dns/node-resolver-9tk9p"
Jul 23 13:40:53.187200 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.186946    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-run-k8s-cni-cncf-io\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.187200 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187005    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-var-lib-cni-bin\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.187200 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187056    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-lib-modules\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.187200 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187068    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-var-lib-cni-bin\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.187200 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187076    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-system-cni-dir\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.187200 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187113    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-os-release\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.187200 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.186991    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0489f3dd-bc92-4289-a74d-ceb26a4ee20e-host\") pod \"node-ca-ddfgb\" (UID: \"0489f3dd-bc92-4289-a74d-ceb26a4ee20e\") " pod="openshift-image-registry/node-ca-ddfgb"
Jul 23 13:40:53.187200 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187140    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-run\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.187200 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187187    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-cnibin\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.187200 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187195    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-os-release\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187143    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-system-cni-dir\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187242    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-run\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187275    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-multus-daemon-config\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187315    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbfkl\" (UniqueName: \"kubernetes.io/projected/640cd623-c590-466c-8c27-be9913ba96c5-kube-api-access-nbfkl\") pod \"iptables-alerter-j625q\" (UID: \"640cd623-c590-466c-8c27-be9913ba96c5\") " pod="openshift-network-operator/iptables-alerter-j625q"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187335    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-sysctl-d\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187332    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-cnibin\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187356    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-multus-conf-dir\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187383    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-multus-conf-dir\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187408    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-multus-socket-dir-parent\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187432    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-var-lib-kubelet\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187462    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c329870-9269-4992-9b11-d3dac8faba35-tmp\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187480    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-sysctl-d\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187502    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-var-lib-kubelet\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187509    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-multus-socket-dir-parent\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187540    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5-tmp-dir\") pod \"node-resolver-9tk9p\" (UID: \"12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5\") " pod="openshift-dns/node-resolver-9tk9p"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187573    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-multus-cni-dir\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187595    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-run-netns\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187626    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-hostroot\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187652    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75fg\" (UniqueName: \"kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg\") pod \"network-check-target-6fh9p\" (UID: \"cf03b6c6-d0e9-4040-91d7-c33ddbe30d03\") " pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187653    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-multus-cni-dir\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187684    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-hostroot\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187685    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-run-netns\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187760    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-etc-kubernetes\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187787    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/640cd623-c590-466c-8c27-be9913ba96c5-iptables-alerter-script\") pod \"iptables-alerter-j625q\" (UID: \"640cd623-c590-466c-8c27-be9913ba96c5\") " pod="openshift-network-operator/iptables-alerter-j625q"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187829    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/640cd623-c590-466c-8c27-be9913ba96c5-host-slash\") pod \"iptables-alerter-j625q\" (UID: \"640cd623-c590-466c-8c27-be9913ba96c5\") " pod="openshift-network-operator/iptables-alerter-j625q"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187868    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5-tmp-dir\") pod \"node-resolver-9tk9p\" (UID: \"12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5\") " pod="openshift-dns/node-resolver-9tk9p"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187874    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-etc-kubernetes\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187878    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-sysconfig\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187914    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-multus-daemon-config\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187928    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/640cd623-c590-466c-8c27-be9913ba96c5-host-slash\") pod \"iptables-alerter-j625q\" (UID: \"640cd623-c590-466c-8c27-be9913ba96c5\") " pod="openshift-network-operator/iptables-alerter-j625q"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187938    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-sysconfig\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187921    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-kubernetes\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187978    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c329870-9269-4992-9b11-d3dac8faba35-etc-tuned\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187985    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-kubernetes\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.187999    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7696\" (UniqueName: \"kubernetes.io/projected/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-kube-api-access-n7696\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.188947 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.188045    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx9nj\" (UniqueName: \"kubernetes.io/projected/7c329870-9269-4992-9b11-d3dac8faba35-kube-api-access-vx9nj\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.188072    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-669cc\" (UniqueName: \"kubernetes.io/projected/0489f3dd-bc92-4289-a74d-ceb26a4ee20e-kube-api-access-669cc\") pod \"node-ca-ddfgb\" (UID: \"0489f3dd-bc92-4289-a74d-ceb26a4ee20e\") " pod="openshift-image-registry/node-ca-ddfgb"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.188436    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-systemd\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.188502    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-run-multus-certs\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.188538    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-modprobe-d\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.188542    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-systemd\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.188567    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-sysctl-conf\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.188600    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-sys\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.188661    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/640cd623-c590-466c-8c27-be9913ba96c5-iptables-alerter-script\") pod \"iptables-alerter-j625q\" (UID: \"640cd623-c590-466c-8c27-be9913ba96c5\") " pod="openshift-network-operator/iptables-alerter-j625q"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.188706    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-modprobe-d\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.188757    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-run-multus-certs\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.188794    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0489f3dd-bc92-4289-a74d-ceb26a4ee20e-serviceca\") pod \"node-ca-ddfgb\" (UID: \"0489f3dd-bc92-4289-a74d-ceb26a4ee20e\") " pod="openshift-image-registry/node-ca-ddfgb"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.188837    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-sys\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.188921    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-var-lib-cni-multus\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.189069    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-host-var-lib-cni-multus\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.189105    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-etc-sysctl-conf\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.189130    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-host\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.189176    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjfzw\" (UniqueName: \"kubernetes.io/projected/12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5-kube-api-access-sjfzw\") pod \"node-resolver-9tk9p\" (UID: \"12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5\") " pod="openshift-dns/node-resolver-9tk9p"
Jul 23 13:40:53.189758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.189392    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0489f3dd-bc92-4289-a74d-ceb26a4ee20e-serviceca\") pod \"node-ca-ddfgb\" (UID: \"0489f3dd-bc92-4289-a74d-ceb26a4ee20e\") " pod="openshift-image-registry/node-ca-ddfgb"
Jul 23 13:40:53.190635 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.189477    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c329870-9269-4992-9b11-d3dac8faba35-host\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.190635 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.190098    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-cni-binary-copy\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.190635 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.190530    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-cni-binary-copy\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.192091 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.191378    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c329870-9269-4992-9b11-d3dac8faba35-etc-tuned\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.192307 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.192288    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c329870-9269-4992-9b11-d3dac8faba35-tmp\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.196580 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:53.196560    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 13:40:53.196580 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:53.196579    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 13:40:53.196729 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:53.196589    2568 projected.go:196] Error preparing data for projected volume kube-api-access-m75fg for pod openshift-network-diagnostics/network-check-target-6fh9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 13:40:53.196729 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:53.196697    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg podName:cf03b6c6-d0e9-4040-91d7-c33ddbe30d03 nodeName:}" failed. No retries permitted until 2026-07-23 13:40:53.69668388 +0000 UTC m=+2.157774903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m75fg" (UniqueName: "kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg") pod "network-check-target-6fh9p" (UID: "cf03b6c6-d0e9-4040-91d7-c33ddbe30d03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 13:40:53.198602 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.198576    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7696\" (UniqueName: \"kubernetes.io/projected/cbcbc9d1-8582-4af0-8390-ba67fe6b1978-kube-api-access-n7696\") pod \"multus-qt824\" (UID: \"cbcbc9d1-8582-4af0-8390-ba67fe6b1978\") " pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.198689 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.198604    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbfkl\" (UniqueName: \"kubernetes.io/projected/640cd623-c590-466c-8c27-be9913ba96c5-kube-api-access-nbfkl\") pod \"iptables-alerter-j625q\" (UID: \"640cd623-c590-466c-8c27-be9913ba96c5\") " pod="openshift-network-operator/iptables-alerter-j625q"
Jul 23 13:40:53.200673 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.200654    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-669cc\" (UniqueName: \"kubernetes.io/projected/0489f3dd-bc92-4289-a74d-ceb26a4ee20e-kube-api-access-669cc\") pod \"node-ca-ddfgb\" (UID: \"0489f3dd-bc92-4289-a74d-ceb26a4ee20e\") " pod="openshift-image-registry/node-ca-ddfgb"
Jul 23 13:40:53.200975 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.200960    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjfzw\" (UniqueName: \"kubernetes.io/projected/12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5-kube-api-access-sjfzw\") pod \"node-resolver-9tk9p\" (UID: \"12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5\") " pod="openshift-dns/node-resolver-9tk9p"
Jul 23 13:40:53.204394 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.204379    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx9nj\" (UniqueName: \"kubernetes.io/projected/7c329870-9269-4992-9b11-d3dac8faba35-kube-api-access-vx9nj\") pod \"tuned-r5649\" (UID: \"7c329870-9269-4992-9b11-d3dac8faba35\") " pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.314790 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.314759    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-44tr7"
Jul 23 13:40:53.336795 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.336769    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:40:53.342749 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.342729    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xvlj5"
Jul 23 13:40:53.362669 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.362644    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2"
Jul 23 13:40:53.378447 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.378426    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ddfgb"
Jul 23 13:40:53.396726 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.396705    2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 13:40:53.396925 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.396904    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qt824"
Jul 23 13:40:53.409724 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.409705    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j625q"
Jul 23 13:40:53.435644 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.435616    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r5649"
Jul 23 13:40:53.439524 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.439504    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9tk9p"
Jul 23 13:40:53.592789 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.592747    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs\") pod \"network-metrics-daemon-88prr\" (UID: \"2a8b633a-9217-4779-89f3-893586369e2e\") " pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:40:53.593124 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.592801    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret\") pod \"global-pull-secret-syncer-fvtk6\" (UID: \"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1\") " pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:40:53.593124 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:53.592904    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 13:40:53.593124 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:53.592955    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret podName:d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1 nodeName:}" failed. No retries permitted until 2026-07-23 13:40:54.592938308 +0000 UTC m=+3.054029328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret") pod "global-pull-secret-syncer-fvtk6" (UID: "d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1") : object "kube-system"/"original-pull-secret" not registered
Jul 23 13:40:53.593124 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:53.592905    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 13:40:53.593124 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:53.593036    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs podName:2a8b633a-9217-4779-89f3-893586369e2e nodeName:}" failed. No retries permitted until 2026-07-23 13:40:54.593021476 +0000 UTC m=+3.054112499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs") pod "network-metrics-daemon-88prr" (UID: "2a8b633a-9217-4779-89f3-893586369e2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 13:40:53.612447 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:53.612416    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de01a743a50264260f9a41b54c7e4a3.slice/crio-f7cee679f6acb1c34d0e1b81a59eb1bc416753e95bfaa1d60b0a025eb5311850 WatchSource:0}: Error finding container f7cee679f6acb1c34d0e1b81a59eb1bc416753e95bfaa1d60b0a025eb5311850: Status 404 returned error can't find the container with id f7cee679f6acb1c34d0e1b81a59eb1bc416753e95bfaa1d60b0a025eb5311850
Jul 23 13:40:53.616988 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.616967    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 13:40:53.688329 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:53.688274    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8ca143_ff44_4040_ba3c_fd70df8f6439.slice/crio-d139bf9853f3134c05fe36c65392c33943e9dc6d1d06a4c68f42aa31421463b4 WatchSource:0}: Error finding container d139bf9853f3134c05fe36c65392c33943e9dc6d1d06a4c68f42aa31421463b4: Status 404 returned error can't find the container with id d139bf9853f3134c05fe36c65392c33943e9dc6d1d06a4c68f42aa31421463b4
Jul 23 13:40:53.689186 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:53.689167    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092f6d96_3452_48f0_a66e_0866fde1c1b6.slice/crio-48d911d7e06ce87fb4358ae0ecfc30c3bf6f7a6599e8c64b08249bd1c9b6cde6 WatchSource:0}: Error finding container 48d911d7e06ce87fb4358ae0ecfc30c3bf6f7a6599e8c64b08249bd1c9b6cde6: Status 404 returned error can't find the container with id 48d911d7e06ce87fb4358ae0ecfc30c3bf6f7a6599e8c64b08249bd1c9b6cde6
Jul 23 13:40:53.717323 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:53.717293    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374f4643_8632_47b5_ae94_e9be6e6d6c4c.slice/crio-a7c8df553715a11f65e9c602ded83ca46f29563bfa79a6d7dc67f5255d2f25c2 WatchSource:0}: Error finding container a7c8df553715a11f65e9c602ded83ca46f29563bfa79a6d7dc67f5255d2f25c2: Status 404 returned error can't find the container with id a7c8df553715a11f65e9c602ded83ca46f29563bfa79a6d7dc67f5255d2f25c2
Jul 23 13:40:53.730122 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:53.730093    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0489f3dd_bc92_4289_a74d_ceb26a4ee20e.slice/crio-019ee8eecb557c173e49bf77822e513fd928527583cdf1a9244b264fdeab896f WatchSource:0}: Error finding container 019ee8eecb557c173e49bf77822e513fd928527583cdf1a9244b264fdeab896f: Status 404 returned error can't find the container with id 019ee8eecb557c173e49bf77822e513fd928527583cdf1a9244b264fdeab896f
Jul 23 13:40:53.730664 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:53.730643    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932faad9_0c93_47e5_a018_b2b4e79aba10.slice/crio-5bf0d8509f6a93dc55e37875c165f902709288f9581547e21968c7f723cc38f8 WatchSource:0}: Error finding container 5bf0d8509f6a93dc55e37875c165f902709288f9581547e21968c7f723cc38f8: Status 404 returned error can't find the container with id 5bf0d8509f6a93dc55e37875c165f902709288f9581547e21968c7f723cc38f8
Jul 23 13:40:53.785994 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:53.785967    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640cd623_c590_466c_8c27_be9913ba96c5.slice/crio-ec25b4dafd0e2f867a3c4e7caf4c5de0b4d36d4f264823d2a25b692bf09d20b6 WatchSource:0}: Error finding container ec25b4dafd0e2f867a3c4e7caf4c5de0b4d36d4f264823d2a25b692bf09d20b6: Status 404 returned error can't find the container with id ec25b4dafd0e2f867a3c4e7caf4c5de0b4d36d4f264823d2a25b692bf09d20b6
Jul 23 13:40:53.794074 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:53.794054    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75fg\" (UniqueName: \"kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg\") pod \"network-check-target-6fh9p\" (UID: \"cf03b6c6-d0e9-4040-91d7-c33ddbe30d03\") " pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:40:53.794185 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:53.794174    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 13:40:53.794228 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:53.794192    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 13:40:53.794228 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:53.794200    2568 projected.go:196] Error preparing data for projected volume kube-api-access-m75fg for pod openshift-network-diagnostics/network-check-target-6fh9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 13:40:53.794286 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:53.794242    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg podName:cf03b6c6-d0e9-4040-91d7-c33ddbe30d03 nodeName:}" failed. No retries permitted until 2026-07-23 13:40:54.794229364 +0000 UTC m=+3.255320383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m75fg" (UniqueName: "kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg") pod "network-check-target-6fh9p" (UID: "cf03b6c6-d0e9-4040-91d7-c33ddbe30d03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 13:40:53.852030 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:53.852003    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcbc9d1_8582_4af0_8390_ba67fe6b1978.slice/crio-ee1c8fae0a8afba5615a89c7dc0f73d52dbdd0a1d7ca12b4a93351b78ff28792 WatchSource:0}: Error finding container ee1c8fae0a8afba5615a89c7dc0f73d52dbdd0a1d7ca12b4a93351b78ff28792: Status 404 returned error can't find the container with id ee1c8fae0a8afba5615a89c7dc0f73d52dbdd0a1d7ca12b4a93351b78ff28792
Jul 23 13:40:53.893745 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:40:53.893720    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c329870_9269_4992_9b11_d3dac8faba35.slice/crio-d74417e2c38aa87f0b6365395e668faebad20d6e1bc9265e4f8c446f5d4bf489 WatchSource:0}: Error finding container d74417e2c38aa87f0b6365395e668faebad20d6e1bc9265e4f8c446f5d4bf489: Status 404 returned error can't find the container with id d74417e2c38aa87f0b6365395e668faebad20d6e1bc9265e4f8c446f5d4bf489
Jul 23 13:40:54.072237 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:54.072143    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 13:35:53 +0000 UTC" deadline="2028-01-20 20:46:37.377921497 +0000 UTC"
Jul 23 13:40:54.072237 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:54.072178    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13111h5m43.305746913s"
Jul 23 13:40:54.146654 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:54.146364    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:40:54.146654 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:54.146505    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:40:54.150927 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:54.150834    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r5649" event={"ID":"7c329870-9269-4992-9b11-d3dac8faba35","Type":"ContainerStarted","Data":"d74417e2c38aa87f0b6365395e668faebad20d6e1bc9265e4f8c446f5d4bf489"}
Jul 23 13:40:54.152142 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:54.152079    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qt824" event={"ID":"cbcbc9d1-8582-4af0-8390-ba67fe6b1978","Type":"ContainerStarted","Data":"ee1c8fae0a8afba5615a89c7dc0f73d52dbdd0a1d7ca12b4a93351b78ff28792"}
Jul 23 13:40:54.153904 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:54.153830    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j625q" event={"ID":"640cd623-c590-466c-8c27-be9913ba96c5","Type":"ContainerStarted","Data":"ec25b4dafd0e2f867a3c4e7caf4c5de0b4d36d4f264823d2a25b692bf09d20b6"}
Jul 23 13:40:54.155111 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:54.155078    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2" event={"ID":"fc8ca143-ff44-4040-ba3c-fd70df8f6439","Type":"ContainerStarted","Data":"d139bf9853f3134c05fe36c65392c33943e9dc6d1d06a4c68f42aa31421463b4"}
Jul 23 13:40:54.156323 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:54.156255    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-125.ec2.internal" event={"ID":"3de01a743a50264260f9a41b54c7e4a3","Type":"ContainerStarted","Data":"f7cee679f6acb1c34d0e1b81a59eb1bc416753e95bfaa1d60b0a025eb5311850"}
Jul 23 13:40:54.157902 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:54.157835    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xvlj5" event={"ID":"932faad9-0c93-47e5-a018-b2b4e79aba10","Type":"ContainerStarted","Data":"5bf0d8509f6a93dc55e37875c165f902709288f9581547e21968c7f723cc38f8"}
Jul 23 13:40:54.159533 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:54.159506    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ddfgb" event={"ID":"0489f3dd-bc92-4289-a74d-ceb26a4ee20e","Type":"ContainerStarted","Data":"019ee8eecb557c173e49bf77822e513fd928527583cdf1a9244b264fdeab896f"}
Jul 23 13:40:54.161218 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:54.161188    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bswct" event={"ID":"374f4643-8632-47b5-ae94-e9be6e6d6c4c","Type":"ContainerStarted","Data":"a7c8df553715a11f65e9c602ded83ca46f29563bfa79a6d7dc67f5255d2f25c2"}
Jul 23 13:40:54.162661 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:54.162638    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44tr7" event={"ID":"092f6d96-3452-48f0-a66e-0866fde1c1b6","Type":"ContainerStarted","Data":"48d911d7e06ce87fb4358ae0ecfc30c3bf6f7a6599e8c64b08249bd1c9b6cde6"}
Jul 23 13:40:54.402716 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:54.402599    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 13:40:54.600868 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:54.600810    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs\") pod \"network-metrics-daemon-88prr\" (UID: \"2a8b633a-9217-4779-89f3-893586369e2e\") " pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:40:54.601071 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:54.600912    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret\") pod \"global-pull-secret-syncer-fvtk6\" (UID: \"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1\") " pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:40:54.601071 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:54.601045    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 13:40:54.601192 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:54.601109    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret podName:d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1 nodeName:}" failed. No retries permitted until 2026-07-23 13:40:56.601090481 +0000 UTC m=+5.062181506 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret") pod "global-pull-secret-syncer-fvtk6" (UID: "d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1") : object "kube-system"/"original-pull-secret" not registered
Jul 23 13:40:54.601511 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:54.601491    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 13:40:54.601589 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:54.601547    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs podName:2a8b633a-9217-4779-89f3-893586369e2e nodeName:}" failed. No retries permitted until 2026-07-23 13:40:56.601531609 +0000 UTC m=+5.062622644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs") pod "network-metrics-daemon-88prr" (UID: "2a8b633a-9217-4779-89f3-893586369e2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 13:40:54.804397 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:54.804290    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75fg\" (UniqueName: \"kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg\") pod \"network-check-target-6fh9p\" (UID: \"cf03b6c6-d0e9-4040-91d7-c33ddbe30d03\") " pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:40:54.813081 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:54.813043    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 13:40:54.813081 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:54.813085    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 13:40:54.813306 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:54.813102    2568 projected.go:196] Error preparing data for projected volume kube-api-access-m75fg for pod openshift-network-diagnostics/network-check-target-6fh9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 13:40:54.813306 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:54.813178    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg podName:cf03b6c6-d0e9-4040-91d7-c33ddbe30d03 nodeName:}" failed. No retries permitted until 2026-07-23 13:40:56.813157178 +0000 UTC m=+5.274248200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m75fg" (UniqueName: "kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg") pod "network-check-target-6fh9p" (UID: "cf03b6c6-d0e9-4040-91d7-c33ddbe30d03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 13:40:55.073112 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:55.072994    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 13:35:53 +0000 UTC" deadline="2028-01-22 03:16:50.265269599 +0000 UTC"
Jul 23 13:40:55.073112 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:55.073060    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13141h35m55.192213091s"
Jul 23 13:40:55.146116 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:55.146079    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:40:55.146293 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:55.146227    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:40:55.146626 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:55.146608    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:40:55.146720 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:55.146702    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:40:56.145616 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:56.145586    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:40:56.146258 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:56.145716    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:40:56.620682 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:56.620649    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret\") pod \"global-pull-secret-syncer-fvtk6\" (UID: \"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1\") " pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:40:56.620885 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:56.620719    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs\") pod \"network-metrics-daemon-88prr\" (UID: \"2a8b633a-9217-4779-89f3-893586369e2e\") " pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:40:56.620885 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:56.620845    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 13:40:56.621014 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:56.620924    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs podName:2a8b633a-9217-4779-89f3-893586369e2e nodeName:}" failed. No retries permitted until 2026-07-23 13:41:00.620903738 +0000 UTC m=+9.081994761 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs") pod "network-metrics-daemon-88prr" (UID: "2a8b633a-9217-4779-89f3-893586369e2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 13:40:56.621308 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:56.621292    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 13:40:56.621399 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:56.621338    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret podName:d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1 nodeName:}" failed. No retries permitted until 2026-07-23 13:41:00.621324743 +0000 UTC m=+9.082415772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret") pod "global-pull-secret-syncer-fvtk6" (UID: "d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1") : object "kube-system"/"original-pull-secret" not registered
Jul 23 13:40:56.821934 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:56.821895    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75fg\" (UniqueName: \"kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg\") pod \"network-check-target-6fh9p\" (UID: \"cf03b6c6-d0e9-4040-91d7-c33ddbe30d03\") " pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:40:56.822110 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:56.822090    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 13:40:56.822172 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:56.822112    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 13:40:56.822172 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:56.822125    2568 projected.go:196] Error preparing data for projected volume kube-api-access-m75fg for pod openshift-network-diagnostics/network-check-target-6fh9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 13:40:56.822265 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:56.822184    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg podName:cf03b6c6-d0e9-4040-91d7-c33ddbe30d03 nodeName:}" failed. No retries permitted until 2026-07-23 13:41:00.822165446 +0000 UTC m=+9.283256484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m75fg" (UniqueName: "kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg") pod "network-check-target-6fh9p" (UID: "cf03b6c6-d0e9-4040-91d7-c33ddbe30d03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 13:40:57.145869 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:57.145821    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:40:57.146340 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:57.145979    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:40:57.146416 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:57.146389    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:40:57.146513 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:57.146478    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:40:58.146502 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:58.145991    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:40:58.146502 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:58.146131    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:40:59.145982 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:59.145944    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:40:59.146167 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:40:59.145949    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:40:59.146167 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:59.146105    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:40:59.146276 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:40:59.146161    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:00.146879 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:00.146783    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:00.149882 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:00.148706    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:00.652376 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:00.652305    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs\") pod \"network-metrics-daemon-88prr\" (UID: \"2a8b633a-9217-4779-89f3-893586369e2e\") " pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:00.652376 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:00.652379    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret\") pod \"global-pull-secret-syncer-fvtk6\" (UID: \"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1\") " pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:00.652634 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:00.652499    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 13:41:00.652634 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:00.652558    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret podName:d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1 nodeName:}" failed. No retries permitted until 2026-07-23 13:41:08.652540185 +0000 UTC m=+17.113631211 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret") pod "global-pull-secret-syncer-fvtk6" (UID: "d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1") : object "kube-system"/"original-pull-secret" not registered
Jul 23 13:41:00.652750 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:00.652636    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 13:41:00.652750 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:00.652671    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs podName:2a8b633a-9217-4779-89f3-893586369e2e nodeName:}" failed. No retries permitted until 2026-07-23 13:41:08.652659137 +0000 UTC m=+17.113750160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs") pod "network-metrics-daemon-88prr" (UID: "2a8b633a-9217-4779-89f3-893586369e2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 13:41:00.854795 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:00.854660    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75fg\" (UniqueName: \"kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg\") pod \"network-check-target-6fh9p\" (UID: \"cf03b6c6-d0e9-4040-91d7-c33ddbe30d03\") " pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:00.854997 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:00.854868    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 13:41:00.854997 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:00.854894    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 13:41:00.854997 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:00.854907    2568 projected.go:196] Error preparing data for projected volume kube-api-access-m75fg for pod openshift-network-diagnostics/network-check-target-6fh9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 13:41:00.854997 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:00.854975    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg podName:cf03b6c6-d0e9-4040-91d7-c33ddbe30d03 nodeName:}" failed. No retries permitted until 2026-07-23 13:41:08.854955999 +0000 UTC m=+17.316047038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m75fg" (UniqueName: "kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg") pod "network-check-target-6fh9p" (UID: "cf03b6c6-d0e9-4040-91d7-c33ddbe30d03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 13:41:01.146061 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:01.145967    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:01.146228 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:01.145980    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:01.146228 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:01.146181    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:01.146355 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:01.146255    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:02.146592 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:02.146279    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:02.146592 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:02.146391    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:03.145483 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:03.145375    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:03.145483 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:03.145412    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:03.145721 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:03.145507    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:03.145721 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:03.145651    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:04.145527 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:04.145491    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:04.145989 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:04.145625    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:05.146000 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:05.145965    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:05.146467 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:05.145965    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:05.146467 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:05.146108    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:05.146467 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:05.146211    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:06.145996 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:06.145959    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:06.146178 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:06.146100    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:07.146170 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:07.146131    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:07.146170 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:07.146166    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:07.146665 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:07.146262    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:07.146665 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:07.146333    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:08.145371 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:08.145338    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:08.145521 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:08.145468    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:08.715774 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:08.715724    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs\") pod \"network-metrics-daemon-88prr\" (UID: \"2a8b633a-9217-4779-89f3-893586369e2e\") " pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:08.716258 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:08.716172    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret\") pod \"global-pull-secret-syncer-fvtk6\" (UID: \"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1\") " pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:08.716341 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:08.716309    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 13:41:08.716425 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:08.716406    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret podName:d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1 nodeName:}" failed. No retries permitted until 2026-07-23 13:41:24.716380368 +0000 UTC m=+33.177471402 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret") pod "global-pull-secret-syncer-fvtk6" (UID: "d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1") : object "kube-system"/"original-pull-secret" not registered
Jul 23 13:41:08.716501 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:08.716425    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 13:41:08.716573 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:08.716557    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs podName:2a8b633a-9217-4779-89f3-893586369e2e nodeName:}" failed. No retries permitted until 2026-07-23 13:41:24.716537383 +0000 UTC m=+33.177628416 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs") pod "network-metrics-daemon-88prr" (UID: "2a8b633a-9217-4779-89f3-893586369e2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 13:41:08.917494 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:08.917459    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75fg\" (UniqueName: \"kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg\") pod \"network-check-target-6fh9p\" (UID: \"cf03b6c6-d0e9-4040-91d7-c33ddbe30d03\") " pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:08.917665 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:08.917586    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 13:41:08.917665 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:08.917611    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 13:41:08.917665 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:08.917627    2568 projected.go:196] Error preparing data for projected volume kube-api-access-m75fg for pod openshift-network-diagnostics/network-check-target-6fh9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 13:41:08.917793 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:08.917687    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg podName:cf03b6c6-d0e9-4040-91d7-c33ddbe30d03 nodeName:}" failed. No retries permitted until 2026-07-23 13:41:24.917667055 +0000 UTC m=+33.378758111 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m75fg" (UniqueName: "kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg") pod "network-check-target-6fh9p" (UID: "cf03b6c6-d0e9-4040-91d7-c33ddbe30d03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 13:41:09.146199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:09.146113    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:09.146199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:09.146143    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:09.146374 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:09.146254    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:09.146374 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:09.146335    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:10.146094 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:10.146056    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:10.146489 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:10.146195    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:11.145251 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:11.145205    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:11.145426 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:11.145262    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:11.145426 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:11.145344    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:11.145426 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:11.145407    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:12.146993 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:12.146959    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:12.147468 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:12.147057    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:13.146322 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:13.146180    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:13.146399 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:13.146231    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:13.146441 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:13.146398    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:13.146487 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:13.146468    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:13.201690 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:13.201660    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-125.ec2.internal" event={"ID":"3de01a743a50264260f9a41b54c7e4a3","Type":"ContainerStarted","Data":"0a7244c0cb479eb8adfb1f307f3c222ba557340ffb3f79d79227c79735fb4243"}
Jul 23 13:41:13.207439 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:13.207383    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9tk9p" event={"ID":"12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5","Type":"ContainerStarted","Data":"742c506be98a33929e8750dfb1b75608626ebf42986d9cc2d5c2a79ef4fc36bf"}
Jul 23 13:41:13.209503 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:13.209324    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal" event={"ID":"08b7d7feb9d6abfd35e59891c87373ae","Type":"ContainerStarted","Data":"52d08fa3395c4c56f365c79fe4afd776eb33a8a805c2ae84d2b8dfe3a5348a76"}
Jul 23 13:41:13.217928 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:13.217504    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r5649" event={"ID":"7c329870-9269-4992-9b11-d3dac8faba35","Type":"ContainerStarted","Data":"915cdbae60fe7aedacd821c9c5da200b4a0e32a156d3e852c3b537576fdb2459"}
Jul 23 13:41:13.218259 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:13.218216    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-125.ec2.internal" podStartSLOduration=21.218198838 podStartE2EDuration="21.218198838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 13:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:13.218161516 +0000 UTC m=+21.679252562" watchObservedRunningTime="2026-07-23 13:41:13.218198838 +0000 UTC m=+21.679289879"
Jul 23 13:41:13.222349 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:13.221928    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qt824" event={"ID":"cbcbc9d1-8582-4af0-8390-ba67fe6b1978","Type":"ContainerStarted","Data":"8c8a5e3c7994816a8ececb4380bdf57a26075678ca2e79f27565fba91747ff04"}
Jul 23 13:41:13.236648 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:13.236607    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r5649" podStartSLOduration=2.221787009 podStartE2EDuration="21.236592866s" totalImagesPullingTime="19.014805857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 13:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:13.236404928 +0000 UTC m=+21.697495970" watchObservedRunningTime="2026-07-23 13:41:13.236592866 +0000 UTC m=+21.697683908"
Jul 23 13:41:13.255316 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:13.255232    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qt824" podStartSLOduration=2.192145851 podStartE2EDuration="21.255211661s" totalImagesPullingTime="19.06306581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 13:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:13.254604411 +0000 UTC m=+21.715695488" watchObservedRunningTime="2026-07-23 13:41:13.255211661 +0000 UTC m=+21.716302707"
Jul 23 13:41:14.146375 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:14.146126    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:14.146550 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:14.146470    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:14.225370 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:14.225333    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ddfgb" event={"ID":"0489f3dd-bc92-4289-a74d-ceb26a4ee20e","Type":"ContainerStarted","Data":"e69d0a3b4366dc2ceb9a9c909a5a3f6911310fc54aa5fee301afc3140cc28da6"}
Jul 23 13:41:14.227089 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:14.227066    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bswct" event={"ID":"374f4643-8632-47b5-ae94-e9be6e6d6c4c","Type":"ContainerStarted","Data":"09d71e88f983dfc3c6eb9bab52cfd44cc7d1fbaf4686fcdfa52731f373002666"}
Jul 23 13:41:14.227196 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:14.227122    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bswct" event={"ID":"374f4643-8632-47b5-ae94-e9be6e6d6c4c","Type":"ContainerStarted","Data":"3cef8ea9be615a790f737930aee8478b88629961b35342e657663d8aa7152378"}
Jul 23 13:41:14.228327 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:14.228305    2568 generic.go:356] "Generic (PLEG): container finished" podID="092f6d96-3452-48f0-a66e-0866fde1c1b6" containerID="3768091a0ecd63f5d9faf14cdf675721efd26cf8b992b30bab2eed74208fba75" exitCode=0
Jul 23 13:41:14.228411 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:14.228386    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44tr7" event={"ID":"092f6d96-3452-48f0-a66e-0866fde1c1b6","Type":"ContainerDied","Data":"3768091a0ecd63f5d9faf14cdf675721efd26cf8b992b30bab2eed74208fba75"}
Jul 23 13:41:14.229810 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:14.229783    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2" event={"ID":"fc8ca143-ff44-4040-ba3c-fd70df8f6439","Type":"ContainerStarted","Data":"9a6e1c59c56ddd7f1d27d12f0e531dd95e1b30781d9c606171f7e256762ea62d"}
Jul 23 13:41:14.231177 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:14.231154    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9tk9p" event={"ID":"12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5","Type":"ContainerStarted","Data":"84c379ce9eb746a693027b231faf950a2cf1d437a504788ab98948266d385583"}
Jul 23 13:41:14.232393 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:14.232372    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xvlj5" event={"ID":"932faad9-0c93-47e5-a018-b2b4e79aba10","Type":"ContainerStarted","Data":"5d2d9b80561c80cb756899f4118b3a643e74a8f4746c291c147bb57319a0b65a"}
Jul 23 13:41:14.242391 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:14.242355    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ddfgb" podStartSLOduration=3.163608714 podStartE2EDuration="22.242338816s" totalImagesPullingTime="19.078730102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 13:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:14.242232201 +0000 UTC m=+22.703323242" watchObservedRunningTime="2026-07-23 13:41:14.242338816 +0000 UTC m=+22.703429857"
Jul 23 13:41:14.257538 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:14.257499    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9tk9p" podStartSLOduration=22.257487549 podStartE2EDuration="22.257487549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 13:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:14.25716201 +0000 UTC m=+22.718253053" watchObservedRunningTime="2026-07-23 13:41:14.257487549 +0000 UTC m=+22.718578591"
Jul 23 13:41:14.300973 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:14.300917    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xvlj5" podStartSLOduration=3.222878858 podStartE2EDuration="22.300897567s" totalImagesPullingTime="19.078018709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:14.300519883 +0000 UTC m=+22.761610924" watchObservedRunningTime="2026-07-23 13:41:14.300897567 +0000 UTC m=+22.761988612"
Jul 23 13:41:14.757334 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:14.757154    2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 23 13:41:15.050305 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:15.050097    2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T13:41:14.757330649Z","UUID":"4e6e37e0-ea2e-4c5d-84dd-c0e52e348243","Handler":null,"Name":"","Endpoint":""}
Jul 23 13:41:15.051919 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:15.051897    2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 23 13:41:15.052039 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:15.051942    2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 23 13:41:15.149614 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:15.149578    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:15.149917 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:15.149626    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:15.149917 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:15.149706    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:15.149917 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:15.149876    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:15.171815 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:15.171791    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9tk9p_12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5/dns-node-resolver/0.log"
Jul 23 13:41:15.236370 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:15.236311    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j625q" event={"ID":"640cd623-c590-466c-8c27-be9913ba96c5","Type":"ContainerStarted","Data":"8b30f5fbc07097bdbf07ce61af6122ff35c303876e148283df24f26c87251dda"}
Jul 23 13:41:15.238405 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:15.238377    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2" event={"ID":"fc8ca143-ff44-4040-ba3c-fd70df8f6439","Type":"ContainerStarted","Data":"4f4e488014f707f622897484eccd822d11b8d9e1393c1eaca5a1b2fee4fd3a0c"}
Jul 23 13:41:15.240025 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:15.240002    2568 generic.go:356] "Generic (PLEG): container finished" podID="08b7d7feb9d6abfd35e59891c87373ae" containerID="c6d28b1945fd8e0e07d8fb2652bbeaeace89449feafb36d78078f929931079a6" exitCode=0
Jul 23 13:41:15.240134 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:15.240088    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal" event={"ID":"08b7d7feb9d6abfd35e59891c87373ae","Type":"ContainerDied","Data":"c6d28b1945fd8e0e07d8fb2652bbeaeace89449feafb36d78078f929931079a6"}
Jul 23 13:41:15.254177 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:15.254139    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j625q" podStartSLOduration=4.191059103 podStartE2EDuration="23.254126568s" totalImagesPullingTime="19.063067465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 13:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:15.253460582 +0000 UTC m=+23.714551624" watchObservedRunningTime="2026-07-23 13:41:15.254126568 +0000 UTC m=+23.715217609"
Jul 23 13:41:16.145774 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:16.145742    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:16.146064 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:16.145906    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:16.244091 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:16.244056    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal" event={"ID":"08b7d7feb9d6abfd35e59891c87373ae","Type":"ContainerStarted","Data":"bf84314b4ef76a8d5e2da78dec6ed2d8681d01c684050908fe34a2e7e184b94b"}
Jul 23 13:41:16.260926 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:16.260882    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-125.ec2.internal" podStartSLOduration=23.260844185 podStartE2EDuration="23.260844185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 13:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:16.260441066 +0000 UTC m=+24.721532120" watchObservedRunningTime="2026-07-23 13:41:16.260844185 +0000 UTC m=+24.721935228"
Jul 23 13:41:16.347666 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:16.347617    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ddfgb_0489f3dd-bc92-4289-a74d-ceb26a4ee20e/node-ca/0.log"
Jul 23 13:41:17.146086 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:17.146014    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:17.146086 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:17.146056    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:17.146425 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:17.146122    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:17.146425 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:17.146253    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:17.252248 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:17.252209    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2" event={"ID":"fc8ca143-ff44-4040-ba3c-fd70df8f6439","Type":"ContainerStarted","Data":"2cbe632616d49759b050f39a596d70f688ccc12079787ce6413a93de892da53e"}
Jul 23 13:41:17.255578 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:17.255546    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bswct" event={"ID":"374f4643-8632-47b5-ae94-e9be6e6d6c4c","Type":"ContainerStarted","Data":"e6de07c1b00990369f240b1acac5b0ed456deeba9e4a8ab941da58ae29dc0fe0"}
Jul 23 13:41:17.255718 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:17.255610    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bswct" event={"ID":"374f4643-8632-47b5-ae94-e9be6e6d6c4c","Type":"ContainerStarted","Data":"66a5c73c807cb7a5741605faae1deb8f435587b8b4418fce36d919f693e3dece"}
Jul 23 13:41:17.255718 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:17.255651    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bswct" event={"ID":"374f4643-8632-47b5-ae94-e9be6e6d6c4c","Type":"ContainerStarted","Data":"bf5c055981a068500d730faeebc0406e89e79f428f46458dd968e58d580f2693"}
Jul 23 13:41:17.255718 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:17.255678    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bswct" event={"ID":"374f4643-8632-47b5-ae94-e9be6e6d6c4c","Type":"ContainerStarted","Data":"99589cabfe4a65e59fb4f2ee41008a844846f0dde461d3892887d2d496b719b9"}
Jul 23 13:41:17.280029 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:17.279987    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xssf2" podStartSLOduration=3.561187587 podStartE2EDuration="25.279974657s" totalImagesPullingTime="21.71878707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 13:40:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:17.279783821 +0000 UTC m=+25.740874864" watchObservedRunningTime="2026-07-23 13:41:17.279974657 +0000 UTC m=+25.741065698"
Jul 23 13:41:18.145820 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:18.145784    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:18.146023 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:18.145936    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:18.785099 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:18.784952    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xvlj5"
Jul 23 13:41:18.786054 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:18.786035    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xvlj5"
Jul 23 13:41:19.145541 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:19.145470    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:19.145674 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:19.145470    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:19.145674 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:19.145581    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:19.145674 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:19.145634    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:19.262038 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:19.262007    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bswct" event={"ID":"374f4643-8632-47b5-ae94-e9be6e6d6c4c","Type":"ContainerStarted","Data":"9f7e2642930743ca3070af2991c6271b8191628ae49d6e66ca88d235cff5b45f"}
Jul 23 13:41:19.263655 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:19.263629    2568 generic.go:356] "Generic (PLEG): container finished" podID="092f6d96-3452-48f0-a66e-0866fde1c1b6" containerID="0e996c452161cd9446acdd021b77c0e6003a7c65f2fd348d1663b9e6afcc1744" exitCode=0
Jul 23 13:41:19.263771 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:19.263708    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44tr7" event={"ID":"092f6d96-3452-48f0-a66e-0866fde1c1b6","Type":"ContainerDied","Data":"0e996c452161cd9446acdd021b77c0e6003a7c65f2fd348d1663b9e6afcc1744"}
Jul 23 13:41:19.263873 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:19.263840    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xvlj5"
Jul 23 13:41:19.264251 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:19.264237    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xvlj5"
Jul 23 13:41:20.145797 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:20.145715    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:20.146253 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:20.145829    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:21.146226 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:21.146011    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:21.146763 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:21.146017    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:21.146763 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:21.146268    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:21.146763 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:21.146373    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:21.270841 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:21.270809    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bswct" event={"ID":"374f4643-8632-47b5-ae94-e9be6e6d6c4c","Type":"ContainerStarted","Data":"f51540c77ba37b27fa2621d13e026f587602458850707c15d2c09f232cb1d723"}
Jul 23 13:41:21.271076 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:21.271058    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:41:21.273029 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:21.273005    2568 generic.go:356] "Generic (PLEG): container finished" podID="092f6d96-3452-48f0-a66e-0866fde1c1b6" containerID="dcc95fb68c77056aead472389f7d144371d08334651675a02f9e52214c5a432d" exitCode=0
Jul 23 13:41:21.273145 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:21.273076    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44tr7" event={"ID":"092f6d96-3452-48f0-a66e-0866fde1c1b6","Type":"ContainerDied","Data":"dcc95fb68c77056aead472389f7d144371d08334651675a02f9e52214c5a432d"}
Jul 23 13:41:21.286914 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:21.286896    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:41:21.298921 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:21.298876    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bswct" podStartSLOduration=6.763706168 podStartE2EDuration="29.298843918s" totalImagesPullingTime="22.53513775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 13:40:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:21.298249409 +0000 UTC m=+29.759340450" watchObservedRunningTime="2026-07-23 13:41:21.298843918 +0000 UTC m=+29.759934962"
Jul 23 13:41:22.146687 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:22.146659    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:22.147131 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:22.146732    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:22.280267 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:22.280248    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:41:22.280359 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:22.280327    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:41:22.297292 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:22.297270    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:41:23.145751 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:23.145725    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:23.145751 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:23.145754    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:23.145943 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:23.145836    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:23.145943 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:23.145921    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:23.279146 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:23.279121    2568 generic.go:356] "Generic (PLEG): container finished" podID="092f6d96-3452-48f0-a66e-0866fde1c1b6" containerID="bd3ec46aa6a8ddc3f89a1fe11a5d84eaf644eca164c77b39f1f3f12bbb0e12ab" exitCode=0
Jul 23 13:41:23.279483 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:23.279181    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44tr7" event={"ID":"092f6d96-3452-48f0-a66e-0866fde1c1b6","Type":"ContainerDied","Data":"bd3ec46aa6a8ddc3f89a1fe11a5d84eaf644eca164c77b39f1f3f12bbb0e12ab"}
Jul 23 13:41:24.146131 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:24.146098    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:24.146296 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:24.146208    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:24.732393 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:24.732360    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs\") pod \"network-metrics-daemon-88prr\" (UID: \"2a8b633a-9217-4779-89f3-893586369e2e\") " pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:24.732816 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:24.732430    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret\") pod \"global-pull-secret-syncer-fvtk6\" (UID: \"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1\") " pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:24.732816 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:24.732525    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 13:41:24.732816 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:24.732541    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 13:41:24.732816 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:24.732600    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret podName:d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1 nodeName:}" failed. No retries permitted until 2026-07-23 13:41:56.732581025 +0000 UTC m=+65.193672048 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret") pod "global-pull-secret-syncer-fvtk6" (UID: "d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1") : object "kube-system"/"original-pull-secret" not registered
Jul 23 13:41:24.732816 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:24.732621    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs podName:2a8b633a-9217-4779-89f3-893586369e2e nodeName:}" failed. No retries permitted until 2026-07-23 13:41:56.732611212 +0000 UTC m=+65.193702238 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs") pod "network-metrics-daemon-88prr" (UID: "2a8b633a-9217-4779-89f3-893586369e2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 13:41:24.933390 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:24.933354    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75fg\" (UniqueName: \"kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg\") pod \"network-check-target-6fh9p\" (UID: \"cf03b6c6-d0e9-4040-91d7-c33ddbe30d03\") " pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:24.933574 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:24.933542    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 13:41:24.933574 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:24.933564    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 13:41:24.933574 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:24.933576    2568 projected.go:196] Error preparing data for projected volume kube-api-access-m75fg for pod openshift-network-diagnostics/network-check-target-6fh9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 13:41:24.933698 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:24.933637    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg podName:cf03b6c6-d0e9-4040-91d7-c33ddbe30d03 nodeName:}" failed. No retries permitted until 2026-07-23 13:41:56.933619101 +0000 UTC m=+65.394710133 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m75fg" (UniqueName: "kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg") pod "network-check-target-6fh9p" (UID: "cf03b6c6-d0e9-4040-91d7-c33ddbe30d03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 13:41:25.146026 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:25.145997    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:25.146204 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:25.146135    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:25.146204 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:25.146189    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:25.146328 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:25.146305    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:26.145769 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:26.145736    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:26.146175 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:26.145842    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:27.145230 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:27.145195    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:27.145230 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:27.145214    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:27.145440 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:27.145315    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:27.145494 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:27.145457    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:28.145966 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:28.145938    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:28.146341 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:28.146036    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:28.293297 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:28.293264    2568 generic.go:356] "Generic (PLEG): container finished" podID="092f6d96-3452-48f0-a66e-0866fde1c1b6" containerID="e5005b45db2ddad54376779562ea84a743e23f6d20101d4967e813e6be035f33" exitCode=0
Jul 23 13:41:28.293448 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:28.293342    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44tr7" event={"ID":"092f6d96-3452-48f0-a66e-0866fde1c1b6","Type":"ContainerDied","Data":"e5005b45db2ddad54376779562ea84a743e23f6d20101d4967e813e6be035f33"}
Jul 23 13:41:29.145810 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:29.145783    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:29.145981 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:29.145782    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:29.145981 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:29.145908    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:29.145981 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:29.145944    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:29.298298 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:29.298266    2568 generic.go:356] "Generic (PLEG): container finished" podID="092f6d96-3452-48f0-a66e-0866fde1c1b6" containerID="141bdcb5ce015cb5047ea44e9781ed7ac02e849732f43c5306eaa1dbda76ed1a" exitCode=0
Jul 23 13:41:29.298455 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:29.298328    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44tr7" event={"ID":"092f6d96-3452-48f0-a66e-0866fde1c1b6","Type":"ContainerDied","Data":"141bdcb5ce015cb5047ea44e9781ed7ac02e849732f43c5306eaa1dbda76ed1a"}
Jul 23 13:41:30.145714 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:30.145689    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:30.145882 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:30.145772    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:30.303157 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:30.303124    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44tr7" event={"ID":"092f6d96-3452-48f0-a66e-0866fde1c1b6","Type":"ContainerStarted","Data":"b7a1fd58afbe3ff0a921b3c1b9e706640035446ee170348bdaebe474cef3f062"}
Jul 23 13:41:30.330553 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:30.330511    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-44tr7" podStartSLOduration=8.786353574 podStartE2EDuration="38.330495643s" totalImagesPullingTime="29.544142069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 13:40:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:30.330042248 +0000 UTC m=+38.791133290" watchObservedRunningTime="2026-07-23 13:41:30.330495643 +0000 UTC m=+38.791586685"
Jul 23 13:41:31.145667 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:31.145634    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:31.145824 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:31.145640    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:31.145824 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:31.145731    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:31.145824 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:31.145809    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:32.146380 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:32.146353    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:32.146741 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:32.146430    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:33.145416 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:33.145386    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:33.145577 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:33.145386    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:33.145577 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:33.145481    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:33.145577 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:33.145548    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:34.145907 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:34.145875    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:34.146257 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:34.145992    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:35.145808 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:35.145775    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:35.145994 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:35.145895    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:35.145994 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:35.145947    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:35.146282 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:35.146035    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:36.146127 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:36.146093    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:36.146527 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:36.146207    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:37.146017 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:37.145984    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:37.146147 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:37.146104    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:37.146398 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:37.146154    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:37.146398 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:37.146223    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:38.050670 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:38.050643    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fvtk6"]
Jul 23 13:41:38.050798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:38.050768    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:38.050875 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:38.050841    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:38.054003 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:38.053976    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-88prr"]
Jul 23 13:41:38.054133 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:38.054088    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:38.054227 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:38.054205    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:38.054672 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:38.054651    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6fh9p"]
Jul 23 13:41:38.054769 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:38.054748    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:38.054830 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:38.054812    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:40.145499 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:40.145466    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:40.145499 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:40.145491    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:40.146016 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:40.145591    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88prr" podUID="2a8b633a-9217-4779-89f3-893586369e2e"
Jul 23 13:41:40.146016 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:40.145641    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:40.146016 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:40.145723    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fvtk6" podUID="d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1"
Jul 23 13:41:40.146016 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:40.145760    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6fh9p" podUID="cf03b6c6-d0e9-4040-91d7-c33ddbe30d03"
Jul 23 13:41:41.804678 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.804652    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-125.ec2.internal" event="NodeReady"
Jul 23 13:41:41.805008 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.804747    2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 23 13:41:41.860496 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.860434    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-z9dtb"]
Jul 23 13:41:41.873734 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.873710    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pfcfh"]
Jul 23 13:41:41.873883 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.873845    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:41.876658 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.876637    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 23 13:41:41.876762 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.876637    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5fzgx\""
Jul 23 13:41:41.876762 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.876646    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 23 13:41:41.876762 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.876742    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 23 13:41:41.876925 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.876758    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 23 13:41:41.887837 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.887820    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z9dtb"]
Jul 23 13:41:41.887931 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.887907    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pfcfh"]
Jul 23 13:41:41.887984 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.887974    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pfcfh"
Jul 23 13:41:41.892330 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.892312    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l4fll\""
Jul 23 13:41:41.892471 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.892337    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 23 13:41:41.892471 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.892380    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 23 13:41:41.893471 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.893452    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bgdb2"]
Jul 23 13:41:41.914007 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.913986    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bgdb2"]
Jul 23 13:41:41.914108 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.914058    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bgdb2"
Jul 23 13:41:41.922436 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.922417    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 23 13:41:41.922515 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.922472    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 23 13:41:41.922515 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.922483    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 23 13:41:41.922597 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.922542    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dmxg7\""
Jul 23 13:41:41.955261 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.955244    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2610fed0-d41b-49ba-aa2d-b5238ce82ab7-data-volume\") pod \"insights-runtime-extractor-z9dtb\" (UID: \"2610fed0-d41b-49ba-aa2d-b5238ce82ab7\") " pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:41.955356 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.955270    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x294t\" (UniqueName: \"kubernetes.io/projected/2610fed0-d41b-49ba-aa2d-b5238ce82ab7-kube-api-access-x294t\") pod \"insights-runtime-extractor-z9dtb\" (UID: \"2610fed0-d41b-49ba-aa2d-b5238ce82ab7\") " pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:41.955356 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.955296    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2610fed0-d41b-49ba-aa2d-b5238ce82ab7-crio-socket\") pod \"insights-runtime-extractor-z9dtb\" (UID: \"2610fed0-d41b-49ba-aa2d-b5238ce82ab7\") " pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:41.955441 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.955354    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2610fed0-d41b-49ba-aa2d-b5238ce82ab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z9dtb\" (UID: \"2610fed0-d41b-49ba-aa2d-b5238ce82ab7\") " pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:41.955441 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:41.955384    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2610fed0-d41b-49ba-aa2d-b5238ce82ab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z9dtb\" (UID: \"2610fed0-d41b-49ba-aa2d-b5238ce82ab7\") " pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:42.055839 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.055813    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x294t\" (UniqueName: \"kubernetes.io/projected/2610fed0-d41b-49ba-aa2d-b5238ce82ab7-kube-api-access-x294t\") pod \"insights-runtime-extractor-z9dtb\" (UID: \"2610fed0-d41b-49ba-aa2d-b5238ce82ab7\") " pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:42.055976 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.055844    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5287ae-1a85-49a7-8bff-7a6601b53bc2-config-volume\") pod \"dns-default-pfcfh\" (UID: \"fb5287ae-1a85-49a7-8bff-7a6601b53bc2\") " pod="openshift-dns/dns-default-pfcfh"
Jul 23 13:41:42.055976 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.055881    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7zvl\" (UniqueName: \"kubernetes.io/projected/fb5287ae-1a85-49a7-8bff-7a6601b53bc2-kube-api-access-r7zvl\") pod \"dns-default-pfcfh\" (UID: \"fb5287ae-1a85-49a7-8bff-7a6601b53bc2\") " pod="openshift-dns/dns-default-pfcfh"
Jul 23 13:41:42.055976 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.055934    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb5287ae-1a85-49a7-8bff-7a6601b53bc2-tmp-dir\") pod \"dns-default-pfcfh\" (UID: \"fb5287ae-1a85-49a7-8bff-7a6601b53bc2\") " pod="openshift-dns/dns-default-pfcfh"
Jul 23 13:41:42.056084 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.055975    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qszxk\" (UniqueName: \"kubernetes.io/projected/0f0aec49-2a44-4652-8c41-42471186cfc7-kube-api-access-qszxk\") pod \"ingress-canary-bgdb2\" (UID: \"0f0aec49-2a44-4652-8c41-42471186cfc7\") " pod="openshift-ingress-canary/ingress-canary-bgdb2"
Jul 23 13:41:42.056084 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.056033    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2610fed0-d41b-49ba-aa2d-b5238ce82ab7-crio-socket\") pod \"insights-runtime-extractor-z9dtb\" (UID: \"2610fed0-d41b-49ba-aa2d-b5238ce82ab7\") " pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:42.056084 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.056050    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb5287ae-1a85-49a7-8bff-7a6601b53bc2-metrics-tls\") pod \"dns-default-pfcfh\" (UID: \"fb5287ae-1a85-49a7-8bff-7a6601b53bc2\") " pod="openshift-dns/dns-default-pfcfh"
Jul 23 13:41:42.056181 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.056105    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2610fed0-d41b-49ba-aa2d-b5238ce82ab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z9dtb\" (UID: \"2610fed0-d41b-49ba-aa2d-b5238ce82ab7\") " pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:42.056181 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.056128    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f0aec49-2a44-4652-8c41-42471186cfc7-cert\") pod \"ingress-canary-bgdb2\" (UID: \"0f0aec49-2a44-4652-8c41-42471186cfc7\") " pod="openshift-ingress-canary/ingress-canary-bgdb2"
Jul 23 13:41:42.056181 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.056149    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2610fed0-d41b-49ba-aa2d-b5238ce82ab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z9dtb\" (UID: \"2610fed0-d41b-49ba-aa2d-b5238ce82ab7\") " pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:42.056280 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.056185    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2610fed0-d41b-49ba-aa2d-b5238ce82ab7-data-volume\") pod \"insights-runtime-extractor-z9dtb\" (UID: \"2610fed0-d41b-49ba-aa2d-b5238ce82ab7\") " pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:42.056316 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.056286    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2610fed0-d41b-49ba-aa2d-b5238ce82ab7-crio-socket\") pod \"insights-runtime-extractor-z9dtb\" (UID: \"2610fed0-d41b-49ba-aa2d-b5238ce82ab7\") " pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:42.056642 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.056625    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2610fed0-d41b-49ba-aa2d-b5238ce82ab7-data-volume\") pod \"insights-runtime-extractor-z9dtb\" (UID: \"2610fed0-d41b-49ba-aa2d-b5238ce82ab7\") " pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:42.056779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.056763    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2610fed0-d41b-49ba-aa2d-b5238ce82ab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z9dtb\" (UID: \"2610fed0-d41b-49ba-aa2d-b5238ce82ab7\") " pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:42.060136 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.060105    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2610fed0-d41b-49ba-aa2d-b5238ce82ab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z9dtb\" (UID: \"2610fed0-d41b-49ba-aa2d-b5238ce82ab7\") " pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:42.068291 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.068265    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x294t\" (UniqueName: \"kubernetes.io/projected/2610fed0-d41b-49ba-aa2d-b5238ce82ab7-kube-api-access-x294t\") pod \"insights-runtime-extractor-z9dtb\" (UID: \"2610fed0-d41b-49ba-aa2d-b5238ce82ab7\") " pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:42.146863 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.146795    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:42.147281 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.147259    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:42.147432 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.147414    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:42.150888 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.150869    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 13:41:42.151031 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.150939    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 13:41:42.151031 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.151003    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 13:41:42.151171 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.151004    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pwdgr\""
Jul 23 13:41:42.151171 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.151086    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 13:41:42.151388 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.151374    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5nz2m\""
Jul 23 13:41:42.156442 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.156421    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f0aec49-2a44-4652-8c41-42471186cfc7-cert\") pod \"ingress-canary-bgdb2\" (UID: \"0f0aec49-2a44-4652-8c41-42471186cfc7\") " pod="openshift-ingress-canary/ingress-canary-bgdb2"
Jul 23 13:41:42.156532 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.156465    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5287ae-1a85-49a7-8bff-7a6601b53bc2-config-volume\") pod \"dns-default-pfcfh\" (UID: \"fb5287ae-1a85-49a7-8bff-7a6601b53bc2\") " pod="openshift-dns/dns-default-pfcfh"
Jul 23 13:41:42.156532 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.156484    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7zvl\" (UniqueName: \"kubernetes.io/projected/fb5287ae-1a85-49a7-8bff-7a6601b53bc2-kube-api-access-r7zvl\") pod \"dns-default-pfcfh\" (UID: \"fb5287ae-1a85-49a7-8bff-7a6601b53bc2\") " pod="openshift-dns/dns-default-pfcfh"
Jul 23 13:41:42.156532 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.156507    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb5287ae-1a85-49a7-8bff-7a6601b53bc2-tmp-dir\") pod \"dns-default-pfcfh\" (UID: \"fb5287ae-1a85-49a7-8bff-7a6601b53bc2\") " pod="openshift-dns/dns-default-pfcfh"
Jul 23 13:41:42.156532 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.156531    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qszxk\" (UniqueName: \"kubernetes.io/projected/0f0aec49-2a44-4652-8c41-42471186cfc7-kube-api-access-qszxk\") pod \"ingress-canary-bgdb2\" (UID: \"0f0aec49-2a44-4652-8c41-42471186cfc7\") " pod="openshift-ingress-canary/ingress-canary-bgdb2"
Jul 23 13:41:42.156727 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.156555    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb5287ae-1a85-49a7-8bff-7a6601b53bc2-metrics-tls\") pod \"dns-default-pfcfh\" (UID: \"fb5287ae-1a85-49a7-8bff-7a6601b53bc2\") " pod="openshift-dns/dns-default-pfcfh"
Jul 23 13:41:42.157121 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.157013    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb5287ae-1a85-49a7-8bff-7a6601b53bc2-tmp-dir\") pod \"dns-default-pfcfh\" (UID: \"fb5287ae-1a85-49a7-8bff-7a6601b53bc2\") " pod="openshift-dns/dns-default-pfcfh"
Jul 23 13:41:42.157121 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.157092    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5287ae-1a85-49a7-8bff-7a6601b53bc2-config-volume\") pod \"dns-default-pfcfh\" (UID: \"fb5287ae-1a85-49a7-8bff-7a6601b53bc2\") " pod="openshift-dns/dns-default-pfcfh"
Jul 23 13:41:42.158907 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.158881    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f0aec49-2a44-4652-8c41-42471186cfc7-cert\") pod \"ingress-canary-bgdb2\" (UID: \"0f0aec49-2a44-4652-8c41-42471186cfc7\") " pod="openshift-ingress-canary/ingress-canary-bgdb2"
Jul 23 13:41:42.158907 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.158899    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb5287ae-1a85-49a7-8bff-7a6601b53bc2-metrics-tls\") pod \"dns-default-pfcfh\" (UID: \"fb5287ae-1a85-49a7-8bff-7a6601b53bc2\") " pod="openshift-dns/dns-default-pfcfh"
Jul 23 13:41:42.165530 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.165509    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7zvl\" (UniqueName: \"kubernetes.io/projected/fb5287ae-1a85-49a7-8bff-7a6601b53bc2-kube-api-access-r7zvl\") pod \"dns-default-pfcfh\" (UID: \"fb5287ae-1a85-49a7-8bff-7a6601b53bc2\") " pod="openshift-dns/dns-default-pfcfh"
Jul 23 13:41:42.167967 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.167949    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qszxk\" (UniqueName: \"kubernetes.io/projected/0f0aec49-2a44-4652-8c41-42471186cfc7-kube-api-access-qszxk\") pod \"ingress-canary-bgdb2\" (UID: \"0f0aec49-2a44-4652-8c41-42471186cfc7\") " pod="openshift-ingress-canary/ingress-canary-bgdb2"
Jul 23 13:41:42.183779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.183763    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z9dtb"
Jul 23 13:41:42.195916 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.195895    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pfcfh"
Jul 23 13:41:42.221685 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.221670    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bgdb2"
Jul 23 13:41:42.399982 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.399950    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pfcfh"]
Jul 23 13:41:42.400881 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.400820    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z9dtb"]
Jul 23 13:41:42.402810 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:41:42.402789    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2610fed0_d41b_49ba_aa2d_b5238ce82ab7.slice/crio-df5f6ea1f55add381366ccbba2ec4b16b4efba886b99818777fa9397ef088563 WatchSource:0}: Error finding container df5f6ea1f55add381366ccbba2ec4b16b4efba886b99818777fa9397ef088563: Status 404 returned error can't find the container with id df5f6ea1f55add381366ccbba2ec4b16b4efba886b99818777fa9397ef088563
Jul 23 13:41:42.403428 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:41:42.403402    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5287ae_1a85_49a7_8bff_7a6601b53bc2.slice/crio-5b892ac9345db4db6b881f3c01752f08880bb8617c59f72aead94b4fdff5e0e5 WatchSource:0}: Error finding container 5b892ac9345db4db6b881f3c01752f08880bb8617c59f72aead94b4fdff5e0e5: Status 404 returned error can't find the container with id 5b892ac9345db4db6b881f3c01752f08880bb8617c59f72aead94b4fdff5e0e5
Jul 23 13:41:42.405636 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:42.405590    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bgdb2"]
Jul 23 13:41:42.408279 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:41:42.408258    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0aec49_2a44_4652_8c41_42471186cfc7.slice/crio-528296670b9458ddfccc1bc4d3438e02b5fad14552293b50f9cb1418ba96db55 WatchSource:0}: Error finding container 528296670b9458ddfccc1bc4d3438e02b5fad14552293b50f9cb1418ba96db55: Status 404 returned error can't find the container with id 528296670b9458ddfccc1bc4d3438e02b5fad14552293b50f9cb1418ba96db55
Jul 23 13:41:43.209723 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:43.209430    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-rwml7"]
Jul 23 13:41:43.221839 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:43.221812    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-rwml7"]
Jul 23 13:41:43.221983 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:43.221937    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-rwml7"
Jul 23 13:41:43.225695 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:43.225073    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 23 13:41:43.225695 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:43.225386    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bh7pq\""
Jul 23 13:41:43.333438 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:43.333400    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pfcfh" event={"ID":"fb5287ae-1a85-49a7-8bff-7a6601b53bc2","Type":"ContainerStarted","Data":"5b892ac9345db4db6b881f3c01752f08880bb8617c59f72aead94b4fdff5e0e5"}
Jul 23 13:41:43.334897 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:43.334869    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z9dtb" event={"ID":"2610fed0-d41b-49ba-aa2d-b5238ce82ab7","Type":"ContainerStarted","Data":"df5f6ea1f55add381366ccbba2ec4b16b4efba886b99818777fa9397ef088563"}
Jul 23 13:41:43.336180 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:43.336157    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bgdb2" event={"ID":"0f0aec49-2a44-4652-8c41-42471186cfc7","Type":"ContainerStarted","Data":"528296670b9458ddfccc1bc4d3438e02b5fad14552293b50f9cb1418ba96db55"}
Jul 23 13:41:43.366096 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:43.366068    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e48d72cd-4b5b-4514-9eef-0c599c5ee29f-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-rwml7\" (UID: \"e48d72cd-4b5b-4514-9eef-0c599c5ee29f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-rwml7"
Jul 23 13:41:43.467138 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:43.467019    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e48d72cd-4b5b-4514-9eef-0c599c5ee29f-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-rwml7\" (UID: \"e48d72cd-4b5b-4514-9eef-0c599c5ee29f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-rwml7"
Jul 23 13:41:43.467275 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:43.467163    2568 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 23 13:41:43.467275 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:43.467241    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e48d72cd-4b5b-4514-9eef-0c599c5ee29f-tls-certificates podName:e48d72cd-4b5b-4514-9eef-0c599c5ee29f nodeName:}" failed. No retries permitted until 2026-07-23 13:41:43.967218803 +0000 UTC m=+52.428309838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/e48d72cd-4b5b-4514-9eef-0c599c5ee29f-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-rwml7" (UID: "e48d72cd-4b5b-4514-9eef-0c599c5ee29f") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 23 13:41:43.971718 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:43.971680    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e48d72cd-4b5b-4514-9eef-0c599c5ee29f-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-rwml7\" (UID: \"e48d72cd-4b5b-4514-9eef-0c599c5ee29f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-rwml7"
Jul 23 13:41:43.975524 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:43.975498    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e48d72cd-4b5b-4514-9eef-0c599c5ee29f-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-rwml7\" (UID: \"e48d72cd-4b5b-4514-9eef-0c599c5ee29f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-rwml7"
Jul 23 13:41:44.134996 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:44.134958    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-rwml7"
Jul 23 13:41:44.469904 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:44.469875    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-rwml7"]
Jul 23 13:41:44.473724 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:41:44.473689    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48d72cd_4b5b_4514_9eef_0c599c5ee29f.slice/crio-6262ee471f596d97e13156410e8d213716f213e022cec6f4e7a54e883d44b5a2 WatchSource:0}: Error finding container 6262ee471f596d97e13156410e8d213716f213e022cec6f4e7a54e883d44b5a2: Status 404 returned error can't find the container with id 6262ee471f596d97e13156410e8d213716f213e022cec6f4e7a54e883d44b5a2
Jul 23 13:41:45.347025 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:45.344039    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-rwml7" event={"ID":"e48d72cd-4b5b-4514-9eef-0c599c5ee29f","Type":"ContainerStarted","Data":"6262ee471f596d97e13156410e8d213716f213e022cec6f4e7a54e883d44b5a2"}
Jul 23 13:41:45.348316 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:45.348285    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pfcfh" event={"ID":"fb5287ae-1a85-49a7-8bff-7a6601b53bc2","Type":"ContainerStarted","Data":"de1585280b54083f09646cb81de4f93ac5653c73561f7f262dd9d8d7f91bba14"}
Jul 23 13:41:45.348818 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:45.348739    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pfcfh"
Jul 23 13:41:45.348818 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:45.348780    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pfcfh" event={"ID":"fb5287ae-1a85-49a7-8bff-7a6601b53bc2","Type":"ContainerStarted","Data":"228736f41fa7a17ca0532ff4ae4478e706eff44849bc2b9a45cbc6d74388c5e7"}
Jul 23 13:41:45.369351 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:45.369269    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pfcfh" podStartSLOduration=2.158278126 podStartE2EDuration="4.369251452s" totalImagesPullingTime="2.210973326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:45.368745625 +0000 UTC m=+53.829836680" watchObservedRunningTime="2026-07-23 13:41:45.369251452 +0000 UTC m=+53.830342498"
Jul 23 13:41:46.354173 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.354098    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bgdb2" event={"ID":"0f0aec49-2a44-4652-8c41-42471186cfc7","Type":"ContainerStarted","Data":"313596b70709f5c281737c7292f09cf58a7b61d4da7d37712ff75d42668cdb8f"}
Jul 23 13:41:46.372989 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.372926    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bgdb2" podStartSLOduration=2.016519841 podStartE2EDuration="5.372909979s" totalImagesPullingTime="3.356390138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:46.37229599 +0000 UTC m=+54.833387034" watchObservedRunningTime="2026-07-23 13:41:46.372909979 +0000 UTC m=+54.834000999"
Jul 23 13:41:46.800372 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.800339    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b5d545b8-6sgkg"]
Jul 23 13:41:46.808422 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.808393    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:46.811954 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.811931    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 23 13:41:46.812083 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.811968    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 23 13:41:46.812083 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.811979    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jsh6x\""
Jul 23 13:41:46.812083 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.812039    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 23 13:41:46.812083 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.811969    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 23 13:41:46.812350 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.812331    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 23 13:41:46.812465 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.812391    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 23 13:41:46.812465 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.812433    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 23 13:41:46.816843 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.816806    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b5d545b8-6sgkg"]
Jul 23 13:41:46.998399 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.998356    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-config\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:46.998583 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.998502    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-oauth-config\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:46.998583 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.998537    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-oauth-serving-cert\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:46.998677 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.998638    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmsp2\" (UniqueName: \"kubernetes.io/projected/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-kube-api-access-bmsp2\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:46.998730 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.998693    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-service-ca\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:46.998780 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:46.998739    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-serving-cert\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:47.099562 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:47.099472    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-serving-cert\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:47.099562 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:47.099521    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-config\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:47.099793 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:47.099590    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-oauth-config\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:47.099793 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:47.099613    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-oauth-serving-cert\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:47.099793 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:47.099714    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmsp2\" (UniqueName: \"kubernetes.io/projected/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-kube-api-access-bmsp2\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:47.099793 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:47.099760    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-service-ca\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:47.100582 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:47.100529    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-oauth-serving-cert\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:47.100582 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:47.100567    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-service-ca\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:47.100758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:47.100725    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-config\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:47.103913 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:47.103889    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-oauth-config\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:47.104042 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:47.103916    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-serving-cert\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:47.110224 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:47.110196    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmsp2\" (UniqueName: \"kubernetes.io/projected/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-kube-api-access-bmsp2\") pod \"console-6b5d545b8-6sgkg\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") " pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:47.120005 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:47.119979    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:49.308875 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:49.307258    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b5d545b8-6sgkg"]
Jul 23 13:41:49.312367 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:41:49.312332    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8aa511f_8b41_4aa1_89fd_d0a7a7a93d06.slice/crio-612c814a37ec7f67ec14021d276fe3af76330a69dca0e57f99254feec3c89c7d WatchSource:0}: Error finding container 612c814a37ec7f67ec14021d276fe3af76330a69dca0e57f99254feec3c89c7d: Status 404 returned error can't find the container with id 612c814a37ec7f67ec14021d276fe3af76330a69dca0e57f99254feec3c89c7d
Jul 23 13:41:49.363696 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:49.363663    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z9dtb" event={"ID":"2610fed0-d41b-49ba-aa2d-b5238ce82ab7","Type":"ContainerStarted","Data":"0620abde5dd76257619be0324d27931a8b802f547c020c9aaf5f13b963ee5c9c"}
Jul 23 13:41:49.365160 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:49.365135    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-rwml7" event={"ID":"e48d72cd-4b5b-4514-9eef-0c599c5ee29f","Type":"ContainerStarted","Data":"1fb79ec4c75fa6312fa9a4a8a5057b9b42427e04ccdeb59c0d4798fc3e23cb9f"}
Jul 23 13:41:49.365349 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:49.365329    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-rwml7"
Jul 23 13:41:49.366354 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:49.366335    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5d545b8-6sgkg" event={"ID":"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06","Type":"ContainerStarted","Data":"612c814a37ec7f67ec14021d276fe3af76330a69dca0e57f99254feec3c89c7d"}
Jul 23 13:41:49.369974 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:49.369955    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-rwml7"
Jul 23 13:41:49.381639 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:49.381514    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-rwml7" podStartSLOduration=1.682234032 podStartE2EDuration="6.38149957s" totalImagesPullingTime="4.699265538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:49.380954309 +0000 UTC m=+57.842045351" watchObservedRunningTime="2026-07-23 13:41:49.38149957 +0000 UTC m=+57.842590612"
Jul 23 13:41:50.397437 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.397389    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75646675f8-rcfjs"]
Jul 23 13:41:50.401184 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.401160    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.411121 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.410937    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 23 13:41:50.413150 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.412522    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75646675f8-rcfjs"]
Jul 23 13:41:50.530283 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.530242    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-trusted-ca-bundle\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.530460 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.530309    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-config\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.530460 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.530336    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-serving-cert\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.530460 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.530358    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-service-ca\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.530460 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.530436    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-oauth-serving-cert\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.530641 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.530528    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-oauth-config\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.530641 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.530559    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpl9z\" (UniqueName: \"kubernetes.io/projected/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-kube-api-access-gpl9z\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.631106 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.631067    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-oauth-serving-cert\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.631285 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.631129    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-oauth-config\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.631285 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.631155    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpl9z\" (UniqueName: \"kubernetes.io/projected/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-kube-api-access-gpl9z\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.631285 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.631199    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-trusted-ca-bundle\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.631471 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.631446    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-config\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.631548 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.631492    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-serving-cert\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.631548 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.631519    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-service-ca\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.631944 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.631913    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-oauth-serving-cert\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.632069 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.632043    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-trusted-ca-bundle\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.632315 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.632271    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-config\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.632315 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.632310    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-service-ca\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.634124 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.634080    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-oauth-config\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.634262 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.634239    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-serving-cert\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.639684 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.639663    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpl9z\" (UniqueName: \"kubernetes.io/projected/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-kube-api-access-gpl9z\") pod \"console-75646675f8-rcfjs\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") " pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.715769 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.715689    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:41:50.877181 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:50.877144    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75646675f8-rcfjs"]
Jul 23 13:41:50.881051 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:41:50.880896    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2e97a2_2ab5_4ab5_99ba_e4240e6dc0b9.slice/crio-93d14414ce8f35e0271a1eff423f85507e60d87497cba6900bce6d0aac315fc4 WatchSource:0}: Error finding container 93d14414ce8f35e0271a1eff423f85507e60d87497cba6900bce6d0aac315fc4: Status 404 returned error can't find the container with id 93d14414ce8f35e0271a1eff423f85507e60d87497cba6900bce6d0aac315fc4
Jul 23 13:41:51.374063 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:51.374022    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75646675f8-rcfjs" event={"ID":"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9","Type":"ContainerStarted","Data":"93d14414ce8f35e0271a1eff423f85507e60d87497cba6900bce6d0aac315fc4"}
Jul 23 13:41:54.292035 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:54.291999    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bswct"
Jul 23 13:41:54.387729 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:54.387645    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z9dtb" event={"ID":"2610fed0-d41b-49ba-aa2d-b5238ce82ab7","Type":"ContainerStarted","Data":"11d9c75249271f667f7e1acded6ed1161868b7a6c3c2352b478cab7620f8a7e0"}
Jul 23 13:41:54.389149 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:54.389125    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5d545b8-6sgkg" event={"ID":"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06","Type":"ContainerStarted","Data":"a040722d9279c291e312864c5d9f3098ba550990ce11aa92a97345ac5f60d63e"}
Jul 23 13:41:54.390503 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:54.390478    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75646675f8-rcfjs" event={"ID":"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9","Type":"ContainerStarted","Data":"86b6d79a5c6a96a0719812b116bf02a841f191acae94c9c51d50ff3493b55247"}
Jul 23 13:41:54.409374 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:54.409323    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b5d545b8-6sgkg" podStartSLOduration=3.6394781959999998 podStartE2EDuration="8.409309546s" totalImagesPullingTime="4.76983135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:54.408723313 +0000 UTC m=+62.869814356" watchObservedRunningTime="2026-07-23 13:41:54.409309546 +0000 UTC m=+62.870400632"
Jul 23 13:41:54.428065 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:54.428024    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75646675f8-rcfjs" podStartSLOduration=1.19485991 podStartE2EDuration="4.428009038s" totalImagesPullingTime="3.233149128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:54.427214979 +0000 UTC m=+62.888306022" watchObservedRunningTime="2026-07-23 13:41:54.428009038 +0000 UTC m=+62.889100074"
Jul 23 13:41:55.356340 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.356299    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pfcfh"
Jul 23 13:41:55.644317 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.644233    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"]
Jul 23 13:41:55.646920 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.646897    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"
Jul 23 13:41:55.650478 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.650453    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 23 13:41:55.650834 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.650767    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 23 13:41:55.650834 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.650819    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 23 13:41:55.651024 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.650953    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 23 13:41:55.651200 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.651174    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 23 13:41:55.651321 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.651274    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7msp8\""
Jul 23 13:41:55.659076 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.659053    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"]
Jul 23 13:41:55.662903 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.662845    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"]
Jul 23 13:41:55.665755 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.665603    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.668874 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.668644    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 23 13:41:55.668874 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.668674    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-p5r6s\""
Jul 23 13:41:55.668874 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.668740    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 23 13:41:55.669089 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.668847    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/16adce2b-eee7-4f2a-9041-32ff625ba0ec-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.669089 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.668912    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16adce2b-eee7-4f2a-9041-32ff625ba0ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.669089 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.668933    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/16adce2b-eee7-4f2a-9041-32ff625ba0ec-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.669089 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.669024    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83ec07d6-e691-40d8-a261-9194df7ef9da-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-m7swl\" (UID: \"83ec07d6-e691-40d8-a261-9194df7ef9da\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"
Jul 23 13:41:55.669089 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.669041    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 23 13:41:55.669089 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.669058    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83ec07d6-e691-40d8-a261-9194df7ef9da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-m7swl\" (UID: \"83ec07d6-e691-40d8-a261-9194df7ef9da\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"
Jul 23 13:41:55.669526 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.669128    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16adce2b-eee7-4f2a-9041-32ff625ba0ec-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.669526 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.669156    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjdjc\" (UniqueName: \"kubernetes.io/projected/16adce2b-eee7-4f2a-9041-32ff625ba0ec-kube-api-access-vjdjc\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.669526 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.669188    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/16adce2b-eee7-4f2a-9041-32ff625ba0ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.669526 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.669218    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wk4f\" (UniqueName: \"kubernetes.io/projected/83ec07d6-e691-40d8-a261-9194df7ef9da-kube-api-access-4wk4f\") pod \"openshift-state-metrics-79f6958cdf-m7swl\" (UID: \"83ec07d6-e691-40d8-a261-9194df7ef9da\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"
Jul 23 13:41:55.669526 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.669244    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/83ec07d6-e691-40d8-a261-9194df7ef9da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-m7swl\" (UID: \"83ec07d6-e691-40d8-a261-9194df7ef9da\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"
Jul 23 13:41:55.678957 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.678935    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"]
Jul 23 13:41:55.684024 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.683998    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8ngmk"]
Jul 23 13:41:55.687133 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.687117    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.689605 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.689587    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 23 13:41:55.689961 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.689936    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 23 13:41:55.690051 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.689998    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 23 13:41:55.690051 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.690024    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j9cvs\""
Jul 23 13:41:55.769937 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.769904    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wk4f\" (UniqueName: \"kubernetes.io/projected/83ec07d6-e691-40d8-a261-9194df7ef9da-kube-api-access-4wk4f\") pod \"openshift-state-metrics-79f6958cdf-m7swl\" (UID: \"83ec07d6-e691-40d8-a261-9194df7ef9da\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"
Jul 23 13:41:55.770093 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.769943    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-metrics-client-ca\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.770093 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.769972    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/83ec07d6-e691-40d8-a261-9194df7ef9da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-m7swl\" (UID: \"83ec07d6-e691-40d8-a261-9194df7ef9da\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"
Jul 23 13:41:55.770093 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770023    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwclg\" (UniqueName: \"kubernetes.io/projected/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-kube-api-access-kwclg\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.770093 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:55.770084    2568 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 23 13:41:55.770093 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770086    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/16adce2b-eee7-4f2a-9041-32ff625ba0ec-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.770348 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:55.770142    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83ec07d6-e691-40d8-a261-9194df7ef9da-openshift-state-metrics-tls podName:83ec07d6-e691-40d8-a261-9194df7ef9da nodeName:}" failed. No retries permitted until 2026-07-23 13:41:56.270121912 +0000 UTC m=+64.731212933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/83ec07d6-e691-40d8-a261-9194df7ef9da-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-m7swl" (UID: "83ec07d6-e691-40d8-a261-9194df7ef9da") : secret "openshift-state-metrics-tls" not found
Jul 23 13:41:55.770348 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770162    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-node-exporter-tls\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.770348 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770202    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16adce2b-eee7-4f2a-9041-32ff625ba0ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.770348 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770235    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/16adce2b-eee7-4f2a-9041-32ff625ba0ec-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.770348 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770280    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.770348 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770311    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-node-exporter-accelerators-collector-config\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.770643 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770350    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83ec07d6-e691-40d8-a261-9194df7ef9da-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-m7swl\" (UID: \"83ec07d6-e691-40d8-a261-9194df7ef9da\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"
Jul 23 13:41:55.770643 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770399    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83ec07d6-e691-40d8-a261-9194df7ef9da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-m7swl\" (UID: \"83ec07d6-e691-40d8-a261-9194df7ef9da\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"
Jul 23 13:41:55.770643 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770442    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-node-exporter-textfile\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.770643 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770614    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-node-exporter-wtmp\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.770643 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770630    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/16adce2b-eee7-4f2a-9041-32ff625ba0ec-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.770919 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770647    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-root\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.770919 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770692    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-sys\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.770919 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770730    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16adce2b-eee7-4f2a-9041-32ff625ba0ec-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.770919 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770756    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjdjc\" (UniqueName: \"kubernetes.io/projected/16adce2b-eee7-4f2a-9041-32ff625ba0ec-kube-api-access-vjdjc\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.770919 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.770790    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/16adce2b-eee7-4f2a-9041-32ff625ba0ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.771164 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.771086    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83ec07d6-e691-40d8-a261-9194df7ef9da-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-m7swl\" (UID: \"83ec07d6-e691-40d8-a261-9194df7ef9da\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"
Jul 23 13:41:55.771513 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.771489    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/16adce2b-eee7-4f2a-9041-32ff625ba0ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.771646 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.771532    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16adce2b-eee7-4f2a-9041-32ff625ba0ec-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.773054 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.773031    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16adce2b-eee7-4f2a-9041-32ff625ba0ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.773282 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.773261    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83ec07d6-e691-40d8-a261-9194df7ef9da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-m7swl\" (UID: \"83ec07d6-e691-40d8-a261-9194df7ef9da\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"
Jul 23 13:41:55.773508 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.773489    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/16adce2b-eee7-4f2a-9041-32ff625ba0ec-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.779016 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.778975    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wk4f\" (UniqueName: \"kubernetes.io/projected/83ec07d6-e691-40d8-a261-9194df7ef9da-kube-api-access-4wk4f\") pod \"openshift-state-metrics-79f6958cdf-m7swl\" (UID: \"83ec07d6-e691-40d8-a261-9194df7ef9da\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"
Jul 23 13:41:55.780319 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.780286    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjdjc\" (UniqueName: \"kubernetes.io/projected/16adce2b-eee7-4f2a-9041-32ff625ba0ec-kube-api-access-vjdjc\") pod \"kube-state-metrics-56c5594674-mlvrs\" (UID: \"16adce2b-eee7-4f2a-9041-32ff625ba0ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.872199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.871736    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-node-exporter-textfile\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.872199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.871826    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-node-exporter-wtmp\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.872199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.871873    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-root\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.872199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.871902    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-sys\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.872199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.871955    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-metrics-client-ca\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.872199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.872004    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwclg\" (UniqueName: \"kubernetes.io/projected/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-kube-api-access-kwclg\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.872199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.872052    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-node-exporter-tls\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.872199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.872106    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.872729 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.872500    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-node-exporter-accelerators-collector-config\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.873017 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.872909    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-node-exporter-textfile\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.873017 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.873015    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-root\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.873193 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.873140    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-node-exporter-wtmp\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.878456 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.873768    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-metrics-client-ca\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.878456 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.873811    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-sys\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.878456 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.874202    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-node-exporter-accelerators-collector-config\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.879889 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.879845    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-node-exporter-tls\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.881109 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.881079    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.884565 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.884542    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwclg\" (UniqueName: \"kubernetes.io/projected/5e59d57f-280a-42f0-bd4d-15eedfefcdf9-kube-api-access-kwclg\") pod \"node-exporter-8ngmk\" (UID: \"5e59d57f-280a-42f0-bd4d-15eedfefcdf9\") " pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:55.977279 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.977191    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"
Jul 23 13:41:55.997753 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:55.997724    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8ngmk"
Jul 23 13:41:56.109508 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.109456    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-mlvrs"]
Jul 23 13:41:56.277042 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.277003    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/83ec07d6-e691-40d8-a261-9194df7ef9da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-m7swl\" (UID: \"83ec07d6-e691-40d8-a261-9194df7ef9da\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"
Jul 23 13:41:56.279620 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.279596    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/83ec07d6-e691-40d8-a261-9194df7ef9da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-m7swl\" (UID: \"83ec07d6-e691-40d8-a261-9194df7ef9da\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"
Jul 23 13:41:56.559287 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.559207    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"
Jul 23 13:41:56.724783 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.724753    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 13:41:56.729591 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.729570    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.731921 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.731898    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 23 13:41:56.732045 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.731938    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 23 13:41:56.732157 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.732140    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 23 13:41:56.732236 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.732153    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 23 13:41:56.732236 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.732171    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 23 13:41:56.732333 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.732285    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 23 13:41:56.732333 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.732326    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4vl4l\""
Jul 23 13:41:56.732578 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.732562    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 23 13:41:56.732624 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.732608    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 23 13:41:56.732675 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.732570    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 23 13:41:56.743158 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.743135    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 13:41:56.781582 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.781551    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-web-config\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.781733 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.781609    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3a062ea-0034-4eca-957d-05f4bcf25b62-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.781733 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.781653    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tt5n\" (UniqueName: \"kubernetes.io/projected/f3a062ea-0034-4eca-957d-05f4bcf25b62-kube-api-access-8tt5n\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.781892 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.781727    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.781892 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.781764    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret\") pod \"global-pull-secret-syncer-fvtk6\" (UID: \"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1\") " pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:56.781892 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.781805    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.781892 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.781844    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3a062ea-0034-4eca-957d-05f4bcf25b62-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.782112 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.781908    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs\") pod \"network-metrics-daemon-88prr\" (UID: \"2a8b633a-9217-4779-89f3-893586369e2e\") " pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:56.782112 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.781942    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.782112 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.781969    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.782112 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.781991    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3a062ea-0034-4eca-957d-05f4bcf25b62-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.782112 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.782031    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3a062ea-0034-4eca-957d-05f4bcf25b62-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.782112 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.782051    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3a062ea-0034-4eca-957d-05f4bcf25b62-config-out\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.782112 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.782080    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.782112 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.782110    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.784755 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.784734    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 13:41:56.784882 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.784774    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 13:41:56.794665 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.794639    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1-original-pull-secret\") pod \"global-pull-secret-syncer-fvtk6\" (UID: \"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1\") " pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:56.794821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.794801    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8b633a-9217-4779-89f3-893586369e2e-metrics-certs\") pod \"network-metrics-daemon-88prr\" (UID: \"2a8b633a-9217-4779-89f3-893586369e2e\") " pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:56.862680 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.862605    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pwdgr\""
Jul 23 13:41:56.870731 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.870707    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88prr"
Jul 23 13:41:56.882282 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.882257    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fvtk6"
Jul 23 13:41:56.882498 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.882474    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.882600 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.882512    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.882600 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.882536    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3a062ea-0034-4eca-957d-05f4bcf25b62-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.882600 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.882574    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3a062ea-0034-4eca-957d-05f4bcf25b62-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.882600 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.882598    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3a062ea-0034-4eca-957d-05f4bcf25b62-config-out\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.882808 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.882700    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.882808 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.882754    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.882808 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.882788    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-web-config\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.882984 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.882826    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3a062ea-0034-4eca-957d-05f4bcf25b62-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.882984 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.882884    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tt5n\" (UniqueName: \"kubernetes.io/projected/f3a062ea-0034-4eca-957d-05f4bcf25b62-kube-api-access-8tt5n\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.882984 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.882922    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.882984 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.882955    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.883163 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.883142    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3a062ea-0034-4eca-957d-05f4bcf25b62-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.883386 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:56.883370    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3a062ea-0034-4eca-957d-05f4bcf25b62-alertmanager-trusted-ca-bundle podName:f3a062ea-0034-4eca-957d-05f4bcf25b62 nodeName:}" failed. No retries permitted until 2026-07-23 13:41:57.383350205 +0000 UTC m=+65.844441245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f3a062ea-0034-4eca-957d-05f4bcf25b62-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "f3a062ea-0034-4eca-957d-05f4bcf25b62") : configmap references non-existent config key: ca-bundle.crt
Jul 23 13:41:56.883597 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:56.883518    2568 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 23 13:41:56.883597 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:41:56.883568    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-main-tls podName:f3a062ea-0034-4eca-957d-05f4bcf25b62 nodeName:}" failed. No retries permitted until 2026-07-23 13:41:57.383552612 +0000 UTC m=+65.844643636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "f3a062ea-0034-4eca-957d-05f4bcf25b62") : secret "alertmanager-main-tls" not found
Jul 23 13:41:56.884532 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.883778    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3a062ea-0034-4eca-957d-05f4bcf25b62-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.884532 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.884478    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3a062ea-0034-4eca-957d-05f4bcf25b62-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.885766 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.885741    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3a062ea-0034-4eca-957d-05f4bcf25b62-config-out\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.886845 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.886446    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.886845 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.886804    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.887200 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.887180    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.887393 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.887374    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3a062ea-0034-4eca-957d-05f4bcf25b62-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.887992 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.887974    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-web-config\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.888096 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.888018    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.888155 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.888103    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.903308 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.903281    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tt5n\" (UniqueName: \"kubernetes.io/projected/f3a062ea-0034-4eca-957d-05f4bcf25b62-kube-api-access-8tt5n\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:56.984369 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.984328    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75fg\" (UniqueName: \"kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg\") pod \"network-check-target-6fh9p\" (UID: \"cf03b6c6-d0e9-4040-91d7-c33ddbe30d03\") " pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:56.987423 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.987393    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 13:41:56.996930 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:56.996903    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 13:41:57.008061 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:57.008038    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m75fg\" (UniqueName: \"kubernetes.io/projected/cf03b6c6-d0e9-4040-91d7-c33ddbe30d03-kube-api-access-m75fg\") pod \"network-check-target-6fh9p\" (UID: \"cf03b6c6-d0e9-4040-91d7-c33ddbe30d03\") " pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:57.121065 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:57.120986    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:57.121217 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:57.121076    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:57.126503 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:57.126480    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:57.181131 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:57.181103    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5nz2m\""
Jul 23 13:41:57.188550 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:57.188529    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:41:57.387800 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:57.387711    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3a062ea-0034-4eca-957d-05f4bcf25b62-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:57.387800 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:57.387767    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:57.388679 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:57.388651    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3a062ea-0034-4eca-957d-05f4bcf25b62-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:57.390572 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:57.390541    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:57.405872 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:57.405828    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:41:57.640159 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:57.640085    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:41:57.710919 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:41:57.710875    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16adce2b_eee7_4f2a_9041_32ff625ba0ec.slice/crio-d215707326e8e6f3af027aa1a6ef45985ad57c04ac51aabbebae992afdf26c58 WatchSource:0}: Error finding container d215707326e8e6f3af027aa1a6ef45985ad57c04ac51aabbebae992afdf26c58: Status 404 returned error can't find the container with id d215707326e8e6f3af027aa1a6ef45985ad57c04ac51aabbebae992afdf26c58
Jul 23 13:41:57.733748 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:41:57.733701    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e59d57f_280a_42f0_bd4d_15eedfefcdf9.slice/crio-951278b26e83debff93a54276c996b6d676ec4673bc6d60279e38ee7bde3cad3 WatchSource:0}: Error finding container 951278b26e83debff93a54276c996b6d676ec4673bc6d60279e38ee7bde3cad3: Status 404 returned error can't find the container with id 951278b26e83debff93a54276c996b6d676ec4673bc6d60279e38ee7bde3cad3
Jul 23 13:41:57.898986 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:57.898873    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fvtk6"]
Jul 23 13:41:57.901419 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:41:57.901393    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f1fe9a_e7d6_4f58_b01e_e821f8e945b1.slice/crio-8901e72130ecc1b24f287f54ebf1ee1be80b6dbb74415fa2eb19696c62c5a9b8 WatchSource:0}: Error finding container 8901e72130ecc1b24f287f54ebf1ee1be80b6dbb74415fa2eb19696c62c5a9b8: Status 404 returned error can't find the container with id 8901e72130ecc1b24f287f54ebf1ee1be80b6dbb74415fa2eb19696c62c5a9b8
Jul 23 13:41:58.126064 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:58.126041    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6fh9p"]
Jul 23 13:41:58.128494 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:41:58.128451    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf03b6c6_d0e9_4040_91d7_c33ddbe30d03.slice/crio-781c951f6e0c7031999b0e90bfa1f5cfbfbaa955fb36d85ee10cb6ca9e3ec5d5 WatchSource:0}: Error finding container 781c951f6e0c7031999b0e90bfa1f5cfbfbaa955fb36d85ee10cb6ca9e3ec5d5: Status 404 returned error can't find the container with id 781c951f6e0c7031999b0e90bfa1f5cfbfbaa955fb36d85ee10cb6ca9e3ec5d5
Jul 23 13:41:58.129548 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:58.129458    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl"]
Jul 23 13:41:58.130552 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:58.130534    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-88prr"]
Jul 23 13:41:58.132294 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:41:58.132270    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8b633a_9217_4779_89f3_893586369e2e.slice/crio-97ed21af2e04fcd7e9aa8f36bd46b7ee8c8f3106d9bdd50536fa8f01748e22e2 WatchSource:0}: Error finding container 97ed21af2e04fcd7e9aa8f36bd46b7ee8c8f3106d9bdd50536fa8f01748e22e2: Status 404 returned error can't find the container with id 97ed21af2e04fcd7e9aa8f36bd46b7ee8c8f3106d9bdd50536fa8f01748e22e2
Jul 23 13:41:58.132939 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:41:58.132917    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ec07d6_e691_40d8_a261_9194df7ef9da.slice/crio-bb5be7520c130292e45efdbb5c6578fa2d972fd52502665663a66386fe3a97f7 WatchSource:0}: Error finding container bb5be7520c130292e45efdbb5c6578fa2d972fd52502665663a66386fe3a97f7: Status 404 returned error can't find the container with id bb5be7520c130292e45efdbb5c6578fa2d972fd52502665663a66386fe3a97f7
Jul 23 13:41:58.141828 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:58.141710    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 13:41:58.144420 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:41:58.144387    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a062ea_0034_4eca_957d_05f4bcf25b62.slice/crio-f1e09a2fb8b0b7fcfcd411d067a531c1cf76c1ba47750ad526e5d829511768f8 WatchSource:0}: Error finding container f1e09a2fb8b0b7fcfcd411d067a531c1cf76c1ba47750ad526e5d829511768f8: Status 404 returned error can't find the container with id f1e09a2fb8b0b7fcfcd411d067a531c1cf76c1ba47750ad526e5d829511768f8
Jul 23 13:41:58.409626 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:58.409553    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a062ea-0034-4eca-957d-05f4bcf25b62","Type":"ContainerStarted","Data":"f1e09a2fb8b0b7fcfcd411d067a531c1cf76c1ba47750ad526e5d829511768f8"}
Jul 23 13:41:58.410748 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:58.410721    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6fh9p" event={"ID":"cf03b6c6-d0e9-4040-91d7-c33ddbe30d03","Type":"ContainerStarted","Data":"781c951f6e0c7031999b0e90bfa1f5cfbfbaa955fb36d85ee10cb6ca9e3ec5d5"}
Jul 23 13:41:58.412036 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:58.412005    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fvtk6" event={"ID":"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1","Type":"ContainerStarted","Data":"8901e72130ecc1b24f287f54ebf1ee1be80b6dbb74415fa2eb19696c62c5a9b8"}
Jul 23 13:41:58.413308 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:58.413274    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8ngmk" event={"ID":"5e59d57f-280a-42f0-bd4d-15eedfefcdf9","Type":"ContainerStarted","Data":"951278b26e83debff93a54276c996b6d676ec4673bc6d60279e38ee7bde3cad3"}
Jul 23 13:41:58.415227 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:58.415198    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl" event={"ID":"83ec07d6-e691-40d8-a261-9194df7ef9da","Type":"ContainerStarted","Data":"aca175cbd583f39006a81b1f56e646febcb27f97a1c11d27d6b7c93847e9a9dd"}
Jul 23 13:41:58.415317 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:58.415245    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl" event={"ID":"83ec07d6-e691-40d8-a261-9194df7ef9da","Type":"ContainerStarted","Data":"2d8a4864f019236cf3703d6ee90abd96ad7e74fb518398f3ad1b446644001b7e"}
Jul 23 13:41:58.415317 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:58.415273    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl" event={"ID":"83ec07d6-e691-40d8-a261-9194df7ef9da","Type":"ContainerStarted","Data":"bb5be7520c130292e45efdbb5c6578fa2d972fd52502665663a66386fe3a97f7"}
Jul 23 13:41:58.416565 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:58.416539    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-88prr" event={"ID":"2a8b633a-9217-4779-89f3-893586369e2e","Type":"ContainerStarted","Data":"97ed21af2e04fcd7e9aa8f36bd46b7ee8c8f3106d9bdd50536fa8f01748e22e2"}
Jul 23 13:41:58.418834 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:58.418809    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z9dtb" event={"ID":"2610fed0-d41b-49ba-aa2d-b5238ce82ab7","Type":"ContainerStarted","Data":"5a3919e71aa39bc99b407a5129bf99c0707ebeb913a3a75c0f0ac472dd28c3a4"}
Jul 23 13:41:58.420136 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:58.420116    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs" event={"ID":"16adce2b-eee7-4f2a-9041-32ff625ba0ec","Type":"ContainerStarted","Data":"d215707326e8e6f3af027aa1a6ef45985ad57c04ac51aabbebae992afdf26c58"}
Jul 23 13:41:58.438817 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:41:58.438755    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-z9dtb" podStartSLOduration=2.066783786 podStartE2EDuration="17.438738242s" totalImagesPullingTime="15.371954456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 13:41:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:41:58.438140145 +0000 UTC m=+66.899231189" watchObservedRunningTime="2026-07-23 13:41:58.438738242 +0000 UTC m=+66.899829284"
Jul 23 13:42:00.441540 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:00.441508    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz"]
Jul 23 13:42:00.445390 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:00.445367    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz"
Jul 23 13:42:00.448379 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:00.448166    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 23 13:42:00.448379 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:00.448178    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2hq67\""
Jul 23 13:42:00.453920 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:00.453842    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz"]
Jul 23 13:42:00.527223 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:00.527086    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b70ebbe6-05ea-4095-bb0e-e39c2709ea86-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-cw7mz\" (UID: \"b70ebbe6-05ea-4095-bb0e-e39c2709ea86\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz"
Jul 23 13:42:00.527223 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:00.527170    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tf2w\" (UniqueName: \"kubernetes.io/projected/b70ebbe6-05ea-4095-bb0e-e39c2709ea86-kube-api-access-5tf2w\") pod \"monitoring-plugin-7b7b7bc8bf-cw7mz\" (UID: \"b70ebbe6-05ea-4095-bb0e-e39c2709ea86\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz"
Jul 23 13:42:00.628572 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:00.628542    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tf2w\" (UniqueName: \"kubernetes.io/projected/b70ebbe6-05ea-4095-bb0e-e39c2709ea86-kube-api-access-5tf2w\") pod \"monitoring-plugin-7b7b7bc8bf-cw7mz\" (UID: \"b70ebbe6-05ea-4095-bb0e-e39c2709ea86\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz"
Jul 23 13:42:00.628690 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:00.628614    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b70ebbe6-05ea-4095-bb0e-e39c2709ea86-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-cw7mz\" (UID: \"b70ebbe6-05ea-4095-bb0e-e39c2709ea86\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz"
Jul 23 13:42:00.628765 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:42:00.628749    2568 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 23 13:42:00.628844 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:42:00.628825    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b70ebbe6-05ea-4095-bb0e-e39c2709ea86-monitoring-plugin-cert podName:b70ebbe6-05ea-4095-bb0e-e39c2709ea86 nodeName:}" failed. No retries permitted until 2026-07-23 13:42:01.128802068 +0000 UTC m=+69.589893102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/b70ebbe6-05ea-4095-bb0e-e39c2709ea86-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-cw7mz" (UID: "b70ebbe6-05ea-4095-bb0e-e39c2709ea86") : secret "monitoring-plugin-cert" not found
Jul 23 13:42:00.637513 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:00.637485    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tf2w\" (UniqueName: \"kubernetes.io/projected/b70ebbe6-05ea-4095-bb0e-e39c2709ea86-kube-api-access-5tf2w\") pod \"monitoring-plugin-7b7b7bc8bf-cw7mz\" (UID: \"b70ebbe6-05ea-4095-bb0e-e39c2709ea86\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz"
Jul 23 13:42:00.716842 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:00.716801    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:42:00.716961 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:00.716923    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:42:00.724182 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:00.723801    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:42:01.134336 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:01.134296    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b70ebbe6-05ea-4095-bb0e-e39c2709ea86-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-cw7mz\" (UID: \"b70ebbe6-05ea-4095-bb0e-e39c2709ea86\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz"
Jul 23 13:42:01.136962 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:01.136938    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b70ebbe6-05ea-4095-bb0e-e39c2709ea86-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-cw7mz\" (UID: \"b70ebbe6-05ea-4095-bb0e-e39c2709ea86\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz"
Jul 23 13:42:01.358545 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:01.358514    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz"
Jul 23 13:42:01.436454 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:01.436370    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8ngmk" event={"ID":"5e59d57f-280a-42f0-bd4d-15eedfefcdf9","Type":"ContainerStarted","Data":"c99ca7a3687f362dea4e00dcc3548ef017da654bedf5beb889a937c252d7f16c"}
Jul 23 13:42:01.438605 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:01.438577    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl" event={"ID":"83ec07d6-e691-40d8-a261-9194df7ef9da","Type":"ContainerStarted","Data":"5ad25404a6651cb283fbd77ded2a959870c1ede7a2ec6f29c336f3383d07ee79"}
Jul 23 13:42:01.440157 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:01.440126    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-88prr" event={"ID":"2a8b633a-9217-4779-89f3-893586369e2e","Type":"ContainerStarted","Data":"217a3bc4e6b4cd16248aa542ccc12f5079d01be9274506629283a4b08cf41498"}
Jul 23 13:42:01.441791 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:01.441770    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs" event={"ID":"16adce2b-eee7-4f2a-9041-32ff625ba0ec","Type":"ContainerStarted","Data":"59ce94949f600eca9661e919eefaff00039c8ad8fa8b5be74b24db44fb843eb6"}
Jul 23 13:42:01.443571 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:01.443545    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a062ea-0034-4eca-957d-05f4bcf25b62","Type":"ContainerStarted","Data":"a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886"}
Jul 23 13:42:01.448366 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:01.448343    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:42:01.494392 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:01.494340    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m7swl" podStartSLOduration=4.17164248 podStartE2EDuration="6.494319304s" totalImagesPullingTime="2.322676824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:42:01.492882641 +0000 UTC m=+69.953973683" watchObservedRunningTime="2026-07-23 13:42:01.494319304 +0000 UTC m=+69.955410350"
Jul 23 13:42:01.591642 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:01.591604    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b5d545b8-6sgkg"]
Jul 23 13:42:02.449139 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:02.449100    2568 generic.go:356] "Generic (PLEG): container finished" podID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerID="a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886" exitCode=0
Jul 23 13:42:02.449604 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:02.449179    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a062ea-0034-4eca-957d-05f4bcf25b62","Type":"ContainerDied","Data":"a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886"}
Jul 23 13:42:02.451378 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:02.451346    2568 generic.go:356] "Generic (PLEG): container finished" podID="5e59d57f-280a-42f0-bd4d-15eedfefcdf9" containerID="c99ca7a3687f362dea4e00dcc3548ef017da654bedf5beb889a937c252d7f16c" exitCode=0
Jul 23 13:42:02.451504 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:02.451433    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8ngmk" event={"ID":"5e59d57f-280a-42f0-bd4d-15eedfefcdf9","Type":"ContainerDied","Data":"c99ca7a3687f362dea4e00dcc3548ef017da654bedf5beb889a937c252d7f16c"}
Jul 23 13:42:03.421486 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.421452    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz"]
Jul 23 13:42:03.426821 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:42:03.426789    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70ebbe6_05ea_4095_bb0e_e39c2709ea86.slice/crio-b7ab582cdc134db69190ead5fe6f159470f551f3a9e479513648f59f9bee6a70 WatchSource:0}: Error finding container b7ab582cdc134db69190ead5fe6f159470f551f3a9e479513648f59f9bee6a70: Status 404 returned error can't find the container with id b7ab582cdc134db69190ead5fe6f159470f551f3a9e479513648f59f9bee6a70
Jul 23 13:42:03.460010 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.459655    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6fh9p" event={"ID":"cf03b6c6-d0e9-4040-91d7-c33ddbe30d03","Type":"ContainerStarted","Data":"323beefbf67b610ae9b1a13753dd5d28a859870285ac626b1fd0e6122345d427"}
Jul 23 13:42:03.460010 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.459748    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:42:03.463896 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.463867    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fvtk6" event={"ID":"d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1","Type":"ContainerStarted","Data":"8dd4a519b6931259e6749c124fee58dbc36d6faae51aa32d123cd93f10667f9c"}
Jul 23 13:42:03.467295 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.467265    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8ngmk" event={"ID":"5e59d57f-280a-42f0-bd4d-15eedfefcdf9","Type":"ContainerStarted","Data":"5dc17ea0eb5d450947562d7808bb0cba1f7f9115f081ce099edf3b2b6bf2b19d"}
Jul 23 13:42:03.467421 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.467322    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8ngmk" event={"ID":"5e59d57f-280a-42f0-bd4d-15eedfefcdf9","Type":"ContainerStarted","Data":"775e18d21bc376bd7c738244208479839b885ce13873c3d5dede1b825eb9258f"}
Jul 23 13:42:03.470133 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.470107    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-88prr" event={"ID":"2a8b633a-9217-4779-89f3-893586369e2e","Type":"ContainerStarted","Data":"52948b55a5815ff35b843958dfd6b00379e1f4552768873d0ecc296f60aefb29"}
Jul 23 13:42:03.475363 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.475337    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs" event={"ID":"16adce2b-eee7-4f2a-9041-32ff625ba0ec","Type":"ContainerStarted","Data":"28202c7b16c4e252db22704255c70f2eff2c8455ebe9a03fb1bc60c498e1f953"}
Jul 23 13:42:03.475457 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.475395    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs" event={"ID":"16adce2b-eee7-4f2a-9041-32ff625ba0ec","Type":"ContainerStarted","Data":"62c45ba4ba13e5c1ee534ecaf373750ab1bbdd02740df56af2e4082186fd86bd"}
Jul 23 13:42:03.477298 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.477259    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6fh9p" podStartSLOduration=66.364100889 podStartE2EDuration="1m11.477244736s" totalImagesPullingTime="5.113143847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:42:03.476069439 +0000 UTC m=+71.937160475" watchObservedRunningTime="2026-07-23 13:42:03.477244736 +0000 UTC m=+71.938335778"
Jul 23 13:42:03.478489 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.478465    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz" event={"ID":"b70ebbe6-05ea-4095-bb0e-e39c2709ea86","Type":"ContainerStarted","Data":"b7ab582cdc134db69190ead5fe6f159470f551f3a9e479513648f59f9bee6a70"}
Jul 23 13:42:03.500289 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.500234    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-mlvrs" podStartSLOduration=5.571238815 podStartE2EDuration="8.500216047s" totalImagesPullingTime="2.928977232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:42:03.500191991 +0000 UTC m=+71.961283035" watchObservedRunningTime="2026-07-23 13:42:03.500216047 +0000 UTC m=+71.961307090"
Jul 23 13:42:03.521872 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.521808    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fvtk6" podStartSLOduration=66.182331555 podStartE2EDuration="1m11.521795744s" totalImagesPullingTime="5.339464189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 13:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:42:03.520907193 +0000 UTC m=+71.981998238" watchObservedRunningTime="2026-07-23 13:42:03.521795744 +0000 UTC m=+71.982886786"
Jul 23 13:42:03.546351 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.546309    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8ngmk" podStartSLOduration=5.567277666 podStartE2EDuration="8.546296625s" totalImagesPullingTime="2.864960682s" totalInitContainerRuntime="114.058277ms" isStatefulPod=true podCreationTimestamp="2026-07-23 13:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:42:03.545926916 +0000 UTC m=+72.007017959" watchObservedRunningTime="2026-07-23 13:42:03.546296625 +0000 UTC m=+72.007387666"
Jul 23 13:42:03.578315 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.578265    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-88prr" podStartSLOduration=69.067710806 podStartE2EDuration="1m11.578247628s" totalImagesPullingTime="2.510536822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:42:03.577066108 +0000 UTC m=+72.038157155" watchObservedRunningTime="2026-07-23 13:42:03.578247628 +0000 UTC m=+72.039338671"
Jul 23 13:42:03.631212 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.631179    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54d77cc564-v9wk6"]
Jul 23 13:42:03.633800 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.633770    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.648936 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.648915    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54d77cc564-v9wk6"]
Jul 23 13:42:03.657058 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.657036    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-trusted-ca-bundle\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.657166 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.657071    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tddw2\" (UniqueName: \"kubernetes.io/projected/5b050fd9-b8b1-471f-b826-077cc0650a46-kube-api-access-tddw2\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.657166 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.657097    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-service-ca\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.657166 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.657121    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-oauth-serving-cert\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.657166 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.657143    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b050fd9-b8b1-471f-b826-077cc0650a46-console-serving-cert\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.657166 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.657164    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-console-config\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.657316 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.657181    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b050fd9-b8b1-471f-b826-077cc0650a46-console-oauth-config\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.758436 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.758403    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-service-ca\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.758595 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.758460    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-oauth-serving-cert\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.758595 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.758498    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b050fd9-b8b1-471f-b826-077cc0650a46-console-serving-cert\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.758595 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.758535    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-console-config\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.758595 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.758561    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b050fd9-b8b1-471f-b826-077cc0650a46-console-oauth-config\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.758824 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.758599    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-trusted-ca-bundle\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.758824 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.758635    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tddw2\" (UniqueName: \"kubernetes.io/projected/5b050fd9-b8b1-471f-b826-077cc0650a46-kube-api-access-tddw2\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.759280 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.759254    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-service-ca\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.759280 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.759256    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-console-config\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.759449 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.759359    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-oauth-serving-cert\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.759806 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.759784    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-trusted-ca-bundle\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.761268 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.761248    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b050fd9-b8b1-471f-b826-077cc0650a46-console-serving-cert\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.761386 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.761365    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b050fd9-b8b1-471f-b826-077cc0650a46-console-oauth-config\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.784551 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.784530    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tddw2\" (UniqueName: \"kubernetes.io/projected/5b050fd9-b8b1-471f-b826-077cc0650a46-kube-api-access-tddw2\") pod \"console-54d77cc564-v9wk6\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") " pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:03.943682 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:03.943644    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:04.092080 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:04.092050    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54d77cc564-v9wk6"]
Jul 23 13:42:04.408624 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:42:04.408552    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b050fd9_b8b1_471f_b826_077cc0650a46.slice/crio-29f9bc8dd6f39be13240562c4a70bcd453b5ad4ea14e8d2a5d298a936defbc9d WatchSource:0}: Error finding container 29f9bc8dd6f39be13240562c4a70bcd453b5ad4ea14e8d2a5d298a936defbc9d: Status 404 returned error can't find the container with id 29f9bc8dd6f39be13240562c4a70bcd453b5ad4ea14e8d2a5d298a936defbc9d
Jul 23 13:42:04.483954 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:04.483881    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d77cc564-v9wk6" event={"ID":"5b050fd9-b8b1-471f-b826-077cc0650a46","Type":"ContainerStarted","Data":"29f9bc8dd6f39be13240562c4a70bcd453b5ad4ea14e8d2a5d298a936defbc9d"}
Jul 23 13:42:05.490029 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:05.489952    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz" event={"ID":"b70ebbe6-05ea-4095-bb0e-e39c2709ea86","Type":"ContainerStarted","Data":"7635e38db99294d11893c679b2a43507a219d9aa429b811d6f759f09174e8890"}
Jul 23 13:42:05.490450 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:05.490134    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz"
Jul 23 13:42:05.494339 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:05.494310    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a062ea-0034-4eca-957d-05f4bcf25b62","Type":"ContainerStarted","Data":"e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da"}
Jul 23 13:42:05.494480 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:05.494384    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a062ea-0034-4eca-957d-05f4bcf25b62","Type":"ContainerStarted","Data":"89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844"}
Jul 23 13:42:05.494480 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:05.494423    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a062ea-0034-4eca-957d-05f4bcf25b62","Type":"ContainerStarted","Data":"9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582"}
Jul 23 13:42:05.494480 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:05.494460    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a062ea-0034-4eca-957d-05f4bcf25b62","Type":"ContainerStarted","Data":"079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c"}
Jul 23 13:42:05.494645 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:05.494499    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a062ea-0034-4eca-957d-05f4bcf25b62","Type":"ContainerStarted","Data":"d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872"}
Jul 23 13:42:05.496344 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:05.496312    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d77cc564-v9wk6" event={"ID":"5b050fd9-b8b1-471f-b826-077cc0650a46","Type":"ContainerStarted","Data":"77f417e02b618ad51d62d746a37f8dad55802c1ec5524b5b39133f56c677d296"}
Jul 23 13:42:05.496449 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:05.496390    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz"
Jul 23 13:42:05.512144 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:05.512101    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cw7mz" podStartSLOduration=4.157724684 podStartE2EDuration="5.512086488s" totalImagesPullingTime="1.354361804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:42:05.510459362 +0000 UTC m=+73.971550407" watchObservedRunningTime="2026-07-23 13:42:05.512086488 +0000 UTC m=+73.973177529"
Jul 23 13:42:05.566823 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:05.566772    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54d77cc564-v9wk6" podStartSLOduration=2.566751421 podStartE2EDuration="2.566751421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:42:05.565439187 +0000 UTC m=+74.026530230" watchObservedRunningTime="2026-07-23 13:42:05.566751421 +0000 UTC m=+74.027842464"
Jul 23 13:42:06.504152 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:06.504111    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a062ea-0034-4eca-957d-05f4bcf25b62","Type":"ContainerStarted","Data":"c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581"}
Jul 23 13:42:06.540500 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:06.540439    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.042917892 podStartE2EDuration="10.540418294s" totalImagesPullingTime="5.474661181s" totalInitContainerRuntime="22.839221ms" isStatefulPod=false podCreationTimestamp="2026-07-23 13:41:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:42:06.538010254 +0000 UTC m=+74.999101296" watchObservedRunningTime="2026-07-23 13:42:06.540418294 +0000 UTC m=+75.001509337"
Jul 23 13:42:13.944187 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:13.944061    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:13.944187 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:13.944153    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:13.948798 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:13.948778    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:14.539971 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:14.539941    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:42:14.590658 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:14.590626    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75646675f8-rcfjs"]
Jul 23 13:42:26.614604 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:26.614558    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b5d545b8-6sgkg" podUID="d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06" containerName="console" containerID="cri-o://a040722d9279c291e312864c5d9f3098ba550990ce11aa92a97345ac5f60d63e" gracePeriod=15
Jul 23 13:42:27.393168 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.393143    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:42:27.559238 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.559206    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmsp2\" (UniqueName: \"kubernetes.io/projected/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-kube-api-access-bmsp2\") pod \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") "
Jul 23 13:42:27.559407 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.559246    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-oauth-config\") pod \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") "
Jul 23 13:42:27.559407 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.559277    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-config\") pod \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") "
Jul 23 13:42:27.559407 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.559306    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-oauth-serving-cert\") pod \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") "
Jul 23 13:42:27.559407 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.559351    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-service-ca\") pod \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") "
Jul 23 13:42:27.559407 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.559383    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-serving-cert\") pod \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\" (UID: \"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06\") "
Jul 23 13:42:27.559700 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.559678    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-config" (OuterVolumeSpecName: "console-config") pod "d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06" (UID: "d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:42:27.559756 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.559731    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06" (UID: "d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:42:27.559800 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.559746    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-service-ca" (OuterVolumeSpecName: "service-ca") pod "d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06" (UID: "d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:42:27.561544 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.561519    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06" (UID: "d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:42:27.561645 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.561587    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06" (UID: "d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:42:27.561686 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.561635    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-kube-api-access-bmsp2" (OuterVolumeSpecName: "kube-api-access-bmsp2") pod "d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06" (UID: "d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06"). InnerVolumeSpecName "kube-api-access-bmsp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 13:42:27.584964 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.584943    2568 generic.go:356] "Generic (PLEG): container finished" podID="d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06" containerID="a040722d9279c291e312864c5d9f3098ba550990ce11aa92a97345ac5f60d63e" exitCode=0
Jul 23 13:42:27.585072 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.584987    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5d545b8-6sgkg"
Jul 23 13:42:27.585072 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.585016    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5d545b8-6sgkg" event={"ID":"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06","Type":"ContainerDied","Data":"a040722d9279c291e312864c5d9f3098ba550990ce11aa92a97345ac5f60d63e"}
Jul 23 13:42:27.585072 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.585055    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5d545b8-6sgkg" event={"ID":"d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06","Type":"ContainerDied","Data":"612c814a37ec7f67ec14021d276fe3af76330a69dca0e57f99254feec3c89c7d"}
Jul 23 13:42:27.585171 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.585075    2568 scope.go:117] "RemoveContainer" containerID="a040722d9279c291e312864c5d9f3098ba550990ce11aa92a97345ac5f60d63e"
Jul 23 13:42:27.593417 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.593396    2568 scope.go:117] "RemoveContainer" containerID="a040722d9279c291e312864c5d9f3098ba550990ce11aa92a97345ac5f60d63e"
Jul 23 13:42:27.593677 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:42:27.593655    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a040722d9279c291e312864c5d9f3098ba550990ce11aa92a97345ac5f60d63e\": container with ID starting with a040722d9279c291e312864c5d9f3098ba550990ce11aa92a97345ac5f60d63e not found: ID does not exist" containerID="a040722d9279c291e312864c5d9f3098ba550990ce11aa92a97345ac5f60d63e"
Jul 23 13:42:27.593734 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.593685    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a040722d9279c291e312864c5d9f3098ba550990ce11aa92a97345ac5f60d63e"} err="failed to get container status \"a040722d9279c291e312864c5d9f3098ba550990ce11aa92a97345ac5f60d63e\": rpc error: code = NotFound desc = could not find container \"a040722d9279c291e312864c5d9f3098ba550990ce11aa92a97345ac5f60d63e\": container with ID starting with a040722d9279c291e312864c5d9f3098ba550990ce11aa92a97345ac5f60d63e not found: ID does not exist"
Jul 23 13:42:27.604662 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.604641    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b5d545b8-6sgkg"]
Jul 23 13:42:27.611621 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.611602    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b5d545b8-6sgkg"]
Jul 23 13:42:27.660105 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.660088    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmsp2\" (UniqueName: \"kubernetes.io/projected/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-kube-api-access-bmsp2\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:42:27.660105 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.660105    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-oauth-config\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:42:27.660395 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.660114    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-config\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:42:27.660395 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.660123    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-oauth-serving-cert\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:42:27.660395 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.660131    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-service-ca\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:42:27.660395 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:27.660140    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06-console-serving-cert\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:42:28.150742 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:28.150713    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06" path="/var/lib/kubelet/pods/d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06/volumes"
Jul 23 13:42:34.486566 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:34.486532    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6fh9p"
Jul 23 13:42:39.611449 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.611387    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75646675f8-rcfjs" podUID="2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9" containerName="console" containerID="cri-o://86b6d79a5c6a96a0719812b116bf02a841f191acae94c9c51d50ff3493b55247" gracePeriod=15
Jul 23 13:42:39.851579 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.851556    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:42:39.955103 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.955039    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-oauth-serving-cert\") pod \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") "
Jul 23 13:42:39.955103 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.955088    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-config\") pod \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") "
Jul 23 13:42:39.955293 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.955112    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-trusted-ca-bundle\") pod \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") "
Jul 23 13:42:39.955293 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.955134    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-oauth-config\") pod \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") "
Jul 23 13:42:39.955293 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.955167    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpl9z\" (UniqueName: \"kubernetes.io/projected/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-kube-api-access-gpl9z\") pod \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") "
Jul 23 13:42:39.955293 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.955217    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-service-ca\") pod \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") "
Jul 23 13:42:39.955293 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.955252    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-serving-cert\") pod \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\" (UID: \"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9\") "
Jul 23 13:42:39.955544 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.955354    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9" (UID: "2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:42:39.955611 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.955552    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-config" (OuterVolumeSpecName: "console-config") pod "2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9" (UID: "2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:42:39.955611 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.955559    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-oauth-serving-cert\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:42:39.956079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.956051    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9" (UID: "2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:42:39.956422 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.956394    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-service-ca" (OuterVolumeSpecName: "service-ca") pod "2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9" (UID: "2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:42:39.957423 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.957386    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9" (UID: "2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:42:39.957520 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.957471    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9" (UID: "2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:42:39.957625 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:39.957598    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-kube-api-access-gpl9z" (OuterVolumeSpecName: "kube-api-access-gpl9z") pod "2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9" (UID: "2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9"). InnerVolumeSpecName "kube-api-access-gpl9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 13:42:40.056014 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:40.055988    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpl9z\" (UniqueName: \"kubernetes.io/projected/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-kube-api-access-gpl9z\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:42:40.056014 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:40.056010    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-service-ca\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:42:40.056130 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:40.056019    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-serving-cert\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:42:40.056130 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:40.056028    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-config\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:42:40.056130 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:40.056035    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-trusted-ca-bundle\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:42:40.056130 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:40.056044    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9-console-oauth-config\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:42:40.635709 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:40.635676    2568 generic.go:356] "Generic (PLEG): container finished" podID="2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9" containerID="86b6d79a5c6a96a0719812b116bf02a841f191acae94c9c51d50ff3493b55247" exitCode=0
Jul 23 13:42:40.636182 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:40.635714    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75646675f8-rcfjs" event={"ID":"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9","Type":"ContainerDied","Data":"86b6d79a5c6a96a0719812b116bf02a841f191acae94c9c51d50ff3493b55247"}
Jul 23 13:42:40.636182 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:40.635760    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75646675f8-rcfjs" event={"ID":"2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9","Type":"ContainerDied","Data":"93d14414ce8f35e0271a1eff423f85507e60d87497cba6900bce6d0aac315fc4"}
Jul 23 13:42:40.636182 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:40.635779    2568 scope.go:117] "RemoveContainer" containerID="86b6d79a5c6a96a0719812b116bf02a841f191acae94c9c51d50ff3493b55247"
Jul 23 13:42:40.636182 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:40.635720    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75646675f8-rcfjs"
Jul 23 13:42:40.643449 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:40.643434    2568 scope.go:117] "RemoveContainer" containerID="86b6d79a5c6a96a0719812b116bf02a841f191acae94c9c51d50ff3493b55247"
Jul 23 13:42:40.643717 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:42:40.643698    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b6d79a5c6a96a0719812b116bf02a841f191acae94c9c51d50ff3493b55247\": container with ID starting with 86b6d79a5c6a96a0719812b116bf02a841f191acae94c9c51d50ff3493b55247 not found: ID does not exist" containerID="86b6d79a5c6a96a0719812b116bf02a841f191acae94c9c51d50ff3493b55247"
Jul 23 13:42:40.643776 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:40.643728    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b6d79a5c6a96a0719812b116bf02a841f191acae94c9c51d50ff3493b55247"} err="failed to get container status \"86b6d79a5c6a96a0719812b116bf02a841f191acae94c9c51d50ff3493b55247\": rpc error: code = NotFound desc = could not find container \"86b6d79a5c6a96a0719812b116bf02a841f191acae94c9c51d50ff3493b55247\": container with ID starting with 86b6d79a5c6a96a0719812b116bf02a841f191acae94c9c51d50ff3493b55247 not found: ID does not exist"
Jul 23 13:42:40.652442 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:40.652418    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75646675f8-rcfjs"]
Jul 23 13:42:40.660154 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:40.660130    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75646675f8-rcfjs"]
Jul 23 13:42:42.149836 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:42:42.149801    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9" path="/var/lib/kubelet/pods/2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9/volumes"
Jul 23 13:43:15.947924 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:15.947893    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 13:43:15.948404 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:15.948182    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="alertmanager" containerID="cri-o://d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872" gracePeriod=120
Jul 23 13:43:15.948404 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:15.948255    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="kube-rbac-proxy-metric" containerID="cri-o://e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da" gracePeriod=120
Jul 23 13:43:15.948404 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:15.948289    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="prom-label-proxy" containerID="cri-o://c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581" gracePeriod=120
Jul 23 13:43:15.948404 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:15.948315    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="kube-rbac-proxy-web" containerID="cri-o://9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582" gracePeriod=120
Jul 23 13:43:15.948404 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:15.948355    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="kube-rbac-proxy" containerID="cri-o://89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844" gracePeriod=120
Jul 23 13:43:15.948715 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:15.948402    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="config-reloader" containerID="cri-o://079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c" gracePeriod=120
Jul 23 13:43:16.765936 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:16.765903    2568 generic.go:356] "Generic (PLEG): container finished" podID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerID="c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581" exitCode=0
Jul 23 13:43:16.765936 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:16.765925    2568 generic.go:356] "Generic (PLEG): container finished" podID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerID="89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844" exitCode=0
Jul 23 13:43:16.765936 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:16.765932    2568 generic.go:356] "Generic (PLEG): container finished" podID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerID="079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c" exitCode=0
Jul 23 13:43:16.765936 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:16.765937    2568 generic.go:356] "Generic (PLEG): container finished" podID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerID="d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872" exitCode=0
Jul 23 13:43:16.766256 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:16.765980    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a062ea-0034-4eca-957d-05f4bcf25b62","Type":"ContainerDied","Data":"c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581"}
Jul 23 13:43:16.766256 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:16.766059    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a062ea-0034-4eca-957d-05f4bcf25b62","Type":"ContainerDied","Data":"89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844"}
Jul 23 13:43:16.766256 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:16.766100    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a062ea-0034-4eca-957d-05f4bcf25b62","Type":"ContainerDied","Data":"079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c"}
Jul 23 13:43:16.766256 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:16.766134    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a062ea-0034-4eca-957d-05f4bcf25b62","Type":"ContainerDied","Data":"d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872"}
Jul 23 13:43:17.199870 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.199829    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:17.222991 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.222970    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-cluster-tls-config\") pod \"f3a062ea-0034-4eca-957d-05f4bcf25b62\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") "
Jul 23 13:43:17.223103 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.223006    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-main-tls\") pod \"f3a062ea-0034-4eca-957d-05f4bcf25b62\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") "
Jul 23 13:43:17.223103 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.223033    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tt5n\" (UniqueName: \"kubernetes.io/projected/f3a062ea-0034-4eca-957d-05f4bcf25b62-kube-api-access-8tt5n\") pod \"f3a062ea-0034-4eca-957d-05f4bcf25b62\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") "
Jul 23 13:43:17.223103 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.223068    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3a062ea-0034-4eca-957d-05f4bcf25b62-alertmanager-main-db\") pod \"f3a062ea-0034-4eca-957d-05f4bcf25b62\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") "
Jul 23 13:43:17.223257 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.223103    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy-web\") pod \"f3a062ea-0034-4eca-957d-05f4bcf25b62\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") "
Jul 23 13:43:17.223257 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.223133    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3a062ea-0034-4eca-957d-05f4bcf25b62-tls-assets\") pod \"f3a062ea-0034-4eca-957d-05f4bcf25b62\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") "
Jul 23 13:43:17.223257 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.223166    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f3a062ea-0034-4eca-957d-05f4bcf25b62\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") "
Jul 23 13:43:17.223257 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.223195    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3a062ea-0034-4eca-957d-05f4bcf25b62-config-out\") pod \"f3a062ea-0034-4eca-957d-05f4bcf25b62\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") "
Jul 23 13:43:17.223257 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.223224    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-config-volume\") pod \"f3a062ea-0034-4eca-957d-05f4bcf25b62\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") "
Jul 23 13:43:17.223257 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.223248    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-web-config\") pod \"f3a062ea-0034-4eca-957d-05f4bcf25b62\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") "
Jul 23 13:43:17.223549 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.223281    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3a062ea-0034-4eca-957d-05f4bcf25b62-alertmanager-trusted-ca-bundle\") pod \"f3a062ea-0034-4eca-957d-05f4bcf25b62\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") "
Jul 23 13:43:17.223549 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.223306    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3a062ea-0034-4eca-957d-05f4bcf25b62-metrics-client-ca\") pod \"f3a062ea-0034-4eca-957d-05f4bcf25b62\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") "
Jul 23 13:43:17.223549 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.223337    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy\") pod \"f3a062ea-0034-4eca-957d-05f4bcf25b62\" (UID: \"f3a062ea-0034-4eca-957d-05f4bcf25b62\") "
Jul 23 13:43:17.223751 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.223718    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a062ea-0034-4eca-957d-05f4bcf25b62-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f3a062ea-0034-4eca-957d-05f4bcf25b62" (UID: "f3a062ea-0034-4eca-957d-05f4bcf25b62"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:43:17.223836 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.223746    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a062ea-0034-4eca-957d-05f4bcf25b62-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f3a062ea-0034-4eca-957d-05f4bcf25b62" (UID: "f3a062ea-0034-4eca-957d-05f4bcf25b62"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 13:43:17.224478 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.224398    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a062ea-0034-4eca-957d-05f4bcf25b62-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f3a062ea-0034-4eca-957d-05f4bcf25b62" (UID: "f3a062ea-0034-4eca-957d-05f4bcf25b62"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:43:17.226741 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.226645    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f3a062ea-0034-4eca-957d-05f4bcf25b62" (UID: "f3a062ea-0034-4eca-957d-05f4bcf25b62"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:43:17.226741 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.226675    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a062ea-0034-4eca-957d-05f4bcf25b62-kube-api-access-8tt5n" (OuterVolumeSpecName: "kube-api-access-8tt5n") pod "f3a062ea-0034-4eca-957d-05f4bcf25b62" (UID: "f3a062ea-0034-4eca-957d-05f4bcf25b62"). InnerVolumeSpecName "kube-api-access-8tt5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 13:43:17.227497 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.227459    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f3a062ea-0034-4eca-957d-05f4bcf25b62" (UID: "f3a062ea-0034-4eca-957d-05f4bcf25b62"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:43:17.227636 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.227608    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f3a062ea-0034-4eca-957d-05f4bcf25b62" (UID: "f3a062ea-0034-4eca-957d-05f4bcf25b62"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:43:17.227811 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.227765    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-config-volume" (OuterVolumeSpecName: "config-volume") pod "f3a062ea-0034-4eca-957d-05f4bcf25b62" (UID: "f3a062ea-0034-4eca-957d-05f4bcf25b62"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:43:17.227957 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.227934    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f3a062ea-0034-4eca-957d-05f4bcf25b62" (UID: "f3a062ea-0034-4eca-957d-05f4bcf25b62"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:43:17.228078 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.228058    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a062ea-0034-4eca-957d-05f4bcf25b62-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f3a062ea-0034-4eca-957d-05f4bcf25b62" (UID: "f3a062ea-0034-4eca-957d-05f4bcf25b62"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 13:43:17.228126 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.228097    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a062ea-0034-4eca-957d-05f4bcf25b62-config-out" (OuterVolumeSpecName: "config-out") pod "f3a062ea-0034-4eca-957d-05f4bcf25b62" (UID: "f3a062ea-0034-4eca-957d-05f4bcf25b62"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 13:43:17.231133 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.231115    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f3a062ea-0034-4eca-957d-05f4bcf25b62" (UID: "f3a062ea-0034-4eca-957d-05f4bcf25b62"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:43:17.236778 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.236758    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-web-config" (OuterVolumeSpecName: "web-config") pod "f3a062ea-0034-4eca-957d-05f4bcf25b62" (UID: "f3a062ea-0034-4eca-957d-05f4bcf25b62"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:43:17.324349 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.324255    2568 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3a062ea-0034-4eca-957d-05f4bcf25b62-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:43:17.324349 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.324297    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3a062ea-0034-4eca-957d-05f4bcf25b62-metrics-client-ca\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:43:17.324349 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.324308    2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:43:17.324349 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.324317    2568 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-cluster-tls-config\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:43:17.324349 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.324326    2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-main-tls\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:43:17.324349 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.324335    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tt5n\" (UniqueName: \"kubernetes.io/projected/f3a062ea-0034-4eca-957d-05f4bcf25b62-kube-api-access-8tt5n\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:43:17.324349 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.324342    2568 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3a062ea-0034-4eca-957d-05f4bcf25b62-alertmanager-main-db\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:43:17.324349 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.324350    2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:43:17.324349 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.324360    2568 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3a062ea-0034-4eca-957d-05f4bcf25b62-tls-assets\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:43:17.324713 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.324368    2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:43:17.324713 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.324378    2568 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3a062ea-0034-4eca-957d-05f4bcf25b62-config-out\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:43:17.324713 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.324386    2568 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-config-volume\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:43:17.324713 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.324417    2568 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3a062ea-0034-4eca-957d-05f4bcf25b62-web-config\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:43:17.772495 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.772464    2568 generic.go:356] "Generic (PLEG): container finished" podID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerID="e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da" exitCode=0
Jul 23 13:43:17.772495 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.772491    2568 generic.go:356] "Generic (PLEG): container finished" podID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerID="9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582" exitCode=0
Jul 23 13:43:17.772667 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.772540    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:17.772667 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.772544    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a062ea-0034-4eca-957d-05f4bcf25b62","Type":"ContainerDied","Data":"e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da"}
Jul 23 13:43:17.772667 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.772609    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a062ea-0034-4eca-957d-05f4bcf25b62","Type":"ContainerDied","Data":"9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582"}
Jul 23 13:43:17.772667 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.772639    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a062ea-0034-4eca-957d-05f4bcf25b62","Type":"ContainerDied","Data":"f1e09a2fb8b0b7fcfcd411d067a531c1cf76c1ba47750ad526e5d829511768f8"}
Jul 23 13:43:17.772786 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.772669    2568 scope.go:117] "RemoveContainer" containerID="c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581"
Jul 23 13:43:17.780130 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.780098    2568 scope.go:117] "RemoveContainer" containerID="e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da"
Jul 23 13:43:17.786551 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.786535    2568 scope.go:117] "RemoveContainer" containerID="89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844"
Jul 23 13:43:17.792587 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.792571    2568 scope.go:117] "RemoveContainer" containerID="9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582"
Jul 23 13:43:17.798641 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.798622    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 13:43:17.799047 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.799031    2568 scope.go:117] "RemoveContainer" containerID="079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c"
Jul 23 13:43:17.801091 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.801071    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 13:43:17.805623 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.805608    2568 scope.go:117] "RemoveContainer" containerID="d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872"
Jul 23 13:43:17.812061 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.812044    2568 scope.go:117] "RemoveContainer" containerID="a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886"
Jul 23 13:43:17.818208 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.818194    2568 scope.go:117] "RemoveContainer" containerID="c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581"
Jul 23 13:43:17.818437 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:43:17.818420    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581\": container with ID starting with c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581 not found: ID does not exist" containerID="c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581"
Jul 23 13:43:17.818481 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.818444    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581"} err="failed to get container status \"c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581\": rpc error: code = NotFound desc = could not find container \"c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581\": container with ID starting with c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581 not found: ID does not exist"
Jul 23 13:43:17.818481 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.818460    2568 scope.go:117] "RemoveContainer" containerID="e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da"
Jul 23 13:43:17.818692 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:43:17.818674    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da\": container with ID starting with e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da not found: ID does not exist" containerID="e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da"
Jul 23 13:43:17.818732 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.818699    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da"} err="failed to get container status \"e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da\": rpc error: code = NotFound desc = could not find container \"e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da\": container with ID starting with e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da not found: ID does not exist"
Jul 23 13:43:17.818732 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.818717    2568 scope.go:117] "RemoveContainer" containerID="89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844"
Jul 23 13:43:17.818962 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:43:17.818944    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844\": container with ID starting with 89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844 not found: ID does not exist" containerID="89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844"
Jul 23 13:43:17.819010 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.818966    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844"} err="failed to get container status \"89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844\": rpc error: code = NotFound desc = could not find container \"89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844\": container with ID starting with 89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844 not found: ID does not exist"
Jul 23 13:43:17.819010 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.818978    2568 scope.go:117] "RemoveContainer" containerID="9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582"
Jul 23 13:43:17.819149 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:43:17.819132    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582\": container with ID starting with 9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582 not found: ID does not exist" containerID="9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582"
Jul 23 13:43:17.819189 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.819154    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582"} err="failed to get container status \"9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582\": rpc error: code = NotFound desc = could not find container \"9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582\": container with ID starting with 9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582 not found: ID does not exist"
Jul 23 13:43:17.819189 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.819173    2568 scope.go:117] "RemoveContainer" containerID="079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c"
Jul 23 13:43:17.819388 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:43:17.819374    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c\": container with ID starting with 079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c not found: ID does not exist" containerID="079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c"
Jul 23 13:43:17.819425 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.819391    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c"} err="failed to get container status \"079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c\": rpc error: code = NotFound desc = could not find container \"079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c\": container with ID starting with 079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c not found: ID does not exist"
Jul 23 13:43:17.819425 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.819403    2568 scope.go:117] "RemoveContainer" containerID="d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872"
Jul 23 13:43:17.819593 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:43:17.819580    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872\": container with ID starting with d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872 not found: ID does not exist" containerID="d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872"
Jul 23 13:43:17.819635 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.819596    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872"} err="failed to get container status \"d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872\": rpc error: code = NotFound desc = could not find container \"d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872\": container with ID starting with d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872 not found: ID does not exist"
Jul 23 13:43:17.819635 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.819607    2568 scope.go:117] "RemoveContainer" containerID="a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886"
Jul 23 13:43:17.819785 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:43:17.819768    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886\": container with ID starting with a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886 not found: ID does not exist" containerID="a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886"
Jul 23 13:43:17.819821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.819789    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886"} err="failed to get container status \"a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886\": rpc error: code = NotFound desc = could not find container \"a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886\": container with ID starting with a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886 not found: ID does not exist"
Jul 23 13:43:17.819821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.819800    2568 scope.go:117] "RemoveContainer" containerID="c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581"
Jul 23 13:43:17.820027 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.820009    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581"} err="failed to get container status \"c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581\": rpc error: code = NotFound desc = could not find container \"c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581\": container with ID starting with c13c8be1e9fdc57a4e808b5adfca37694ebba2fb5d3933eca9df424a7ca89581 not found: ID does not exist"
Jul 23 13:43:17.820079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.820028    2568 scope.go:117] "RemoveContainer" containerID="e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da"
Jul 23 13:43:17.820255 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.820237    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da"} err="failed to get container status \"e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da\": rpc error: code = NotFound desc = could not find container \"e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da\": container with ID starting with e300e2b2748abe52e653c173f3ed7d772845ba348cfaeb4f58a48829f7f781da not found: ID does not exist"
Jul 23 13:43:17.820291 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.820257    2568 scope.go:117] "RemoveContainer" containerID="89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844"
Jul 23 13:43:17.820467 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.820451    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844"} err="failed to get container status \"89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844\": rpc error: code = NotFound desc = could not find container \"89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844\": container with ID starting with 89fa99b66626028af5834c370066c519442a4c134cd3a16a38b75de288e7b844 not found: ID does not exist"
Jul 23 13:43:17.820556 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.820468    2568 scope.go:117] "RemoveContainer" containerID="9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582"
Jul 23 13:43:17.820657 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.820639    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582"} err="failed to get container status \"9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582\": rpc error: code = NotFound desc = could not find container \"9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582\": container with ID starting with 9f2499bf013ca11ea26cc6c3415222381bcb0f751fdd2e4ae26386f14343d582 not found: ID does not exist"
Jul 23 13:43:17.820695 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.820659    2568 scope.go:117] "RemoveContainer" containerID="079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c"
Jul 23 13:43:17.820865 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.820835    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c"} err="failed to get container status \"079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c\": rpc error: code = NotFound desc = could not find container \"079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c\": container with ID starting with 079d92cf3332565b5d2e00294ac0060698efc2312bb4dc26e6d72dcbf110154c not found: ID does not exist"
Jul 23 13:43:17.820915 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.820869    2568 scope.go:117] "RemoveContainer" containerID="d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872"
Jul 23 13:43:17.821073 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.821057    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872"} err="failed to get container status \"d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872\": rpc error: code = NotFound desc = could not find container \"d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872\": container with ID starting with d423041c7638a043d315a878bf85b21c9819c6eaf94aee5785d0c84018288872 not found: ID does not exist"
Jul 23 13:43:17.821073 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.821072    2568 scope.go:117] "RemoveContainer" containerID="a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886"
Jul 23 13:43:17.821250 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.821236    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886"} err="failed to get container status \"a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886\": rpc error: code = NotFound desc = could not find container \"a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886\": container with ID starting with a416c1a640374f6ec46be63225c92c2ede82f882a88e33411e46d93bdf942886 not found: ID does not exist"
Jul 23 13:43:17.836370 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836348    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 13:43:17.836593 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836580    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="init-config-reloader"
Jul 23 13:43:17.836634 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836596    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="init-config-reloader"
Jul 23 13:43:17.836634 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836610    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="kube-rbac-proxy-web"
Jul 23 13:43:17.836634 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836616    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="kube-rbac-proxy-web"
Jul 23 13:43:17.836634 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836623    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="prom-label-proxy"
Jul 23 13:43:17.836634 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836632    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="prom-label-proxy"
Jul 23 13:43:17.836779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836642    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="kube-rbac-proxy-metric"
Jul 23 13:43:17.836779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836648    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="kube-rbac-proxy-metric"
Jul 23 13:43:17.836779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836658    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9" containerName="console"
Jul 23 13:43:17.836779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836663    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9" containerName="console"
Jul 23 13:43:17.836779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836669    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06" containerName="console"
Jul 23 13:43:17.836779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836673    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06" containerName="console"
Jul 23 13:43:17.836779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836678    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="config-reloader"
Jul 23 13:43:17.836779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836684    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="config-reloader"
Jul 23 13:43:17.836779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836720    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="alertmanager"
Jul 23 13:43:17.836779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836726    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="alertmanager"
Jul 23 13:43:17.836779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836732    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="kube-rbac-proxy"
Jul 23 13:43:17.836779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836737    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="kube-rbac-proxy"
Jul 23 13:43:17.836779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836777    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e2e97a2-2ab5-4ab5-99ba-e4240e6dc0b9" containerName="console"
Jul 23 13:43:17.837167 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836784    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8aa511f-8b41-4aa1-89fd-d0a7a7a93d06" containerName="console"
Jul 23 13:43:17.837167 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836792    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="prom-label-proxy"
Jul 23 13:43:17.837167 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836799    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="kube-rbac-proxy"
Jul 23 13:43:17.837167 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836804    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="kube-rbac-proxy-web"
Jul 23 13:43:17.837167 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836810    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="kube-rbac-proxy-metric"
Jul 23 13:43:17.837167 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836815    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="alertmanager"
Jul 23 13:43:17.837167 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.836820    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" containerName="config-reloader"
Jul 23 13:43:17.840607 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.840585    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:17.843349 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.843329    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 23 13:43:17.843441 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.843354    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 23 13:43:17.843441 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.843354    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 23 13:43:17.843833 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.843816    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 23 13:43:17.843926 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.843881    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4vl4l\""
Jul 23 13:43:17.844163 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.844151    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 23 13:43:17.844388 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.844358    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 23 13:43:17.844388 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.844377    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 23 13:43:17.844610 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.844593    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 23 13:43:17.849323 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.849307    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 23 13:43:17.863290 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.860645    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 13:43:17.929447 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.929424    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64df05b4-0f98-4400-be4c-9cc80efcdfef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:17.929545 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.929463    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64df05b4-0f98-4400-be4c-9cc80efcdfef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:17.929545 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.929481    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:17.929545 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.929530    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64df05b4-0f98-4400-be4c-9cc80efcdfef-config-out\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:17.929667 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.929565    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-config-volume\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:17.929667 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.929606    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:17.929667 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.929631    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:17.929769 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.929665    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:17.929769 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.929691    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfrfg\" (UniqueName: \"kubernetes.io/projected/64df05b4-0f98-4400-be4c-9cc80efcdfef-kube-api-access-hfrfg\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:17.929769 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.929710    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-web-config\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:17.929769 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.929736    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/64df05b4-0f98-4400-be4c-9cc80efcdfef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:17.929769 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.929756    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:17.929961 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:17.929771    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64df05b4-0f98-4400-be4c-9cc80efcdfef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.030888 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.030827    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/64df05b4-0f98-4400-be4c-9cc80efcdfef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.030888 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.030884    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.031008 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.030907    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64df05b4-0f98-4400-be4c-9cc80efcdfef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.031008 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.030945    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64df05b4-0f98-4400-be4c-9cc80efcdfef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.031008 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.030990    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64df05b4-0f98-4400-be4c-9cc80efcdfef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.031148 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.031012    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.031148 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.031040    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64df05b4-0f98-4400-be4c-9cc80efcdfef-config-out\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.031148 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.031070    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-config-volume\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.031148 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.031096    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.031148 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.031120    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.031382 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.031196    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.031382 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.031241    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfrfg\" (UniqueName: \"kubernetes.io/projected/64df05b4-0f98-4400-be4c-9cc80efcdfef-kube-api-access-hfrfg\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.031382 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.031273    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-web-config\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.031939 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.031916    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64df05b4-0f98-4400-be4c-9cc80efcdfef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.032073 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.031203    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/64df05b4-0f98-4400-be4c-9cc80efcdfef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.032535 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.032507    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64df05b4-0f98-4400-be4c-9cc80efcdfef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.033679 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.033654    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64df05b4-0f98-4400-be4c-9cc80efcdfef-config-out\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.034169 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.034144    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.034263 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.034208    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-web-config\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.034322 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.034290    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.034449 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.034431    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.034975 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.034956    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.035049 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.035009    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64df05b4-0f98-4400-be4c-9cc80efcdfef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.035108 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.035091    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.035628 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.035610    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/64df05b4-0f98-4400-be4c-9cc80efcdfef-config-volume\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.044426 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.044407    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfrfg\" (UniqueName: \"kubernetes.io/projected/64df05b4-0f98-4400-be4c-9cc80efcdfef-kube-api-access-hfrfg\") pod \"alertmanager-main-0\" (UID: \"64df05b4-0f98-4400-be4c-9cc80efcdfef\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.149316 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.149272    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 13:43:18.150312 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.150288    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3a062ea-0034-4eca-957d-05f4bcf25b62" path="/var/lib/kubelet/pods/f3a062ea-0034-4eca-957d-05f4bcf25b62/volumes"
Jul 23 13:43:18.290289 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.290223    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 13:43:18.293608 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:43:18.293584    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64df05b4_0f98_4400_be4c_9cc80efcdfef.slice/crio-2a8947ccc732abde13e08b4259eebde3a17c9ac0d6c03922c5c3968d963ea3cd WatchSource:0}: Error finding container 2a8947ccc732abde13e08b4259eebde3a17c9ac0d6c03922c5c3968d963ea3cd: Status 404 returned error can't find the container with id 2a8947ccc732abde13e08b4259eebde3a17c9ac0d6c03922c5c3968d963ea3cd
Jul 23 13:43:18.777757 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.777724    2568 generic.go:356] "Generic (PLEG): container finished" podID="64df05b4-0f98-4400-be4c-9cc80efcdfef" containerID="1facc8090b69c587d832d3f663a2d2d0bff07745c248dd3d8b8e814fe9100ddf" exitCode=0
Jul 23 13:43:18.777939 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.777764    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df05b4-0f98-4400-be4c-9cc80efcdfef","Type":"ContainerDied","Data":"1facc8090b69c587d832d3f663a2d2d0bff07745c248dd3d8b8e814fe9100ddf"}
Jul 23 13:43:18.777939 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:18.777813    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df05b4-0f98-4400-be4c-9cc80efcdfef","Type":"ContainerStarted","Data":"2a8947ccc732abde13e08b4259eebde3a17c9ac0d6c03922c5c3968d963ea3cd"}
Jul 23 13:43:19.786084 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:19.786048    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df05b4-0f98-4400-be4c-9cc80efcdfef","Type":"ContainerStarted","Data":"0a9b3f51515afbfe723754d9d27127cc3dda6b34fbf7afbd334b9789cc62eccf"}
Jul 23 13:43:19.786473 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:19.786115    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df05b4-0f98-4400-be4c-9cc80efcdfef","Type":"ContainerStarted","Data":"19a59fbb7d0d49fd9b45ab99851e1f0e60f12a6dd512decd2a8fd522f02ad28f"}
Jul 23 13:43:19.786473 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:19.786144    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df05b4-0f98-4400-be4c-9cc80efcdfef","Type":"ContainerStarted","Data":"1894cdcabfc9a875184c1493c2a960c4ede0b70dc9b6a4ead2d5d5911e4a10c0"}
Jul 23 13:43:19.786473 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:19.786169    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df05b4-0f98-4400-be4c-9cc80efcdfef","Type":"ContainerStarted","Data":"87f1748dd7cb16bf993459fa574363c7d59848cb7961aad63cc25efd5f8599d5"}
Jul 23 13:43:19.786473 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:19.786194    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df05b4-0f98-4400-be4c-9cc80efcdfef","Type":"ContainerStarted","Data":"da8e78f2f5fe5ce56225758214e001e890f205d2d0cda7044e477b079592ceef"}
Jul 23 13:43:19.786473 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:19.786221    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64df05b4-0f98-4400-be4c-9cc80efcdfef","Type":"ContainerStarted","Data":"3ceede30a585cf1631bc416c9f0dff3d030c5986cbb7643db5df908bc0b93589"}
Jul 23 13:43:19.821048 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:19.820994    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.801582345 podStartE2EDuration="2.820975081s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.392736ms" isStatefulPod=false podCreationTimestamp="2026-07-23 13:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:43:19.819510623 +0000 UTC m=+148.280601676" watchObservedRunningTime="2026-07-23 13:43:19.820975081 +0000 UTC m=+148.282066120"
Jul 23 13:43:19.987516 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:19.987484    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-76bff58957-wwhfc"]
Jul 23 13:43:19.990223 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:19.990206    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:19.992924 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:19.992897    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 23 13:43:19.993050 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:19.992931    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 23 13:43:19.993050 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:19.992900    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 23 13:43:19.994361 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:19.994343    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 23 13:43:19.994467 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:19.994368    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-d7wgs\""
Jul 23 13:43:19.994467 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:19.994370    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 23 13:43:20.004049 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.004015    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 23 13:43:20.005038 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.005019    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76bff58957-wwhfc"]
Jul 23 13:43:20.043675 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.043590    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d62d918-599a-438a-ac1f-2ba20a97a4f5-metrics-client-ca\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.043675 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.043638    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d62d918-599a-438a-ac1f-2ba20a97a4f5-serving-certs-ca-bundle\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.043894 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.043694    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7d62d918-599a-438a-ac1f-2ba20a97a4f5-telemeter-client-tls\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.043894 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.043732    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92f9f\" (UniqueName: \"kubernetes.io/projected/7d62d918-599a-438a-ac1f-2ba20a97a4f5-kube-api-access-92f9f\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.043894 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.043778    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7d62d918-599a-438a-ac1f-2ba20a97a4f5-secret-telemeter-client\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.043894 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.043826    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7d62d918-599a-438a-ac1f-2ba20a97a4f5-federate-client-tls\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.043894 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.043848    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d62d918-599a-438a-ac1f-2ba20a97a4f5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.044047 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.043900    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d62d918-599a-438a-ac1f-2ba20a97a4f5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.145121 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.145077    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7d62d918-599a-438a-ac1f-2ba20a97a4f5-federate-client-tls\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.145273 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.145132    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d62d918-599a-438a-ac1f-2ba20a97a4f5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.145273 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.145158    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d62d918-599a-438a-ac1f-2ba20a97a4f5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.145402 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.145333    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d62d918-599a-438a-ac1f-2ba20a97a4f5-metrics-client-ca\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.145452 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.145413    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d62d918-599a-438a-ac1f-2ba20a97a4f5-serving-certs-ca-bundle\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.145501 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.145455    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7d62d918-599a-438a-ac1f-2ba20a97a4f5-telemeter-client-tls\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.145501 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.145485    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92f9f\" (UniqueName: \"kubernetes.io/projected/7d62d918-599a-438a-ac1f-2ba20a97a4f5-kube-api-access-92f9f\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.145599 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.145519    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7d62d918-599a-438a-ac1f-2ba20a97a4f5-secret-telemeter-client\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.146206 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.146176    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d62d918-599a-438a-ac1f-2ba20a97a4f5-metrics-client-ca\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.146206 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.146196    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d62d918-599a-438a-ac1f-2ba20a97a4f5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.146819 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.146793    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d62d918-599a-438a-ac1f-2ba20a97a4f5-serving-certs-ca-bundle\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.148105 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.148077    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7d62d918-599a-438a-ac1f-2ba20a97a4f5-secret-telemeter-client\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.148306 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.148280    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d62d918-599a-438a-ac1f-2ba20a97a4f5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.148385 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.148302    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7d62d918-599a-438a-ac1f-2ba20a97a4f5-federate-client-tls\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.148836 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.148817    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7d62d918-599a-438a-ac1f-2ba20a97a4f5-telemeter-client-tls\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.156111 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.156087    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92f9f\" (UniqueName: \"kubernetes.io/projected/7d62d918-599a-438a-ac1f-2ba20a97a4f5-kube-api-access-92f9f\") pod \"telemeter-client-76bff58957-wwhfc\" (UID: \"7d62d918-599a-438a-ac1f-2ba20a97a4f5\") " pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.303484 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.303402    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc"
Jul 23 13:43:20.442711 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.442683    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76bff58957-wwhfc"]
Jul 23 13:43:20.444096 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:43:20.444066    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d62d918_599a_438a_ac1f_2ba20a97a4f5.slice/crio-58bda1eba5fea6e6f8db9fea1a8e3c8e8bf5f12c913a553d88cafa85d6c1e273 WatchSource:0}: Error finding container 58bda1eba5fea6e6f8db9fea1a8e3c8e8bf5f12c913a553d88cafa85d6c1e273: Status 404 returned error can't find the container with id 58bda1eba5fea6e6f8db9fea1a8e3c8e8bf5f12c913a553d88cafa85d6c1e273
Jul 23 13:43:20.791522 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:20.791492    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc" event={"ID":"7d62d918-599a-438a-ac1f-2ba20a97a4f5","Type":"ContainerStarted","Data":"58bda1eba5fea6e6f8db9fea1a8e3c8e8bf5f12c913a553d88cafa85d6c1e273"}
Jul 23 13:43:22.801911 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:22.801881    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc" event={"ID":"7d62d918-599a-438a-ac1f-2ba20a97a4f5","Type":"ContainerStarted","Data":"98f154e0e42582b31b4889053f068c08e3932a86f7ba33148633578ffab91845"}
Jul 23 13:43:22.802293 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:22.801947    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc" event={"ID":"7d62d918-599a-438a-ac1f-2ba20a97a4f5","Type":"ContainerStarted","Data":"c6c4c3506a188c3814cb6cff2bf7b96ca8950cb6e1f3e7468c0e4ef26f101661"}
Jul 23 13:43:22.802293 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:22.801971    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc" event={"ID":"7d62d918-599a-438a-ac1f-2ba20a97a4f5","Type":"ContainerStarted","Data":"64ab0b8b4f4ff4b59851057ed066a5716034ab3942c3ca502460fa69346ccc6b"}
Jul 23 13:43:22.831375 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:22.831324    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-76bff58957-wwhfc" podStartSLOduration=2.017487478 podStartE2EDuration="3.831310961s" totalImagesPullingTime="1.813823483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:43:22.830776039 +0000 UTC m=+151.291867073" watchObservedRunningTime="2026-07-23 13:43:22.831310961 +0000 UTC m=+151.292402002"
Jul 23 13:43:23.969123 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:23.969095    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76d74ffb9-hx6xl"]
Jul 23 13:43:23.971560 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:23.971542    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:23.989199 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:23.989178    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76d74ffb9-hx6xl"]
Jul 23 13:43:24.075919 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.075890    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-trusted-ca-bundle\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.076054 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.075926    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00e32261-75cf-4304-8605-738a3467ac45-console-serving-cert\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.076054 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.075957    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph8q7\" (UniqueName: \"kubernetes.io/projected/00e32261-75cf-4304-8605-738a3467ac45-kube-api-access-ph8q7\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.076054 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.076008    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-console-config\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.076054 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.076036    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00e32261-75cf-4304-8605-738a3467ac45-console-oauth-config\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.076221 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.076082    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-service-ca\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.076221 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.076140    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-oauth-serving-cert\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.177033 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.177003    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph8q7\" (UniqueName: \"kubernetes.io/projected/00e32261-75cf-4304-8605-738a3467ac45-kube-api-access-ph8q7\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.177181 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.177040    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-console-config\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.177181 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.177061    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00e32261-75cf-4304-8605-738a3467ac45-console-oauth-config\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.177181 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.177085    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-service-ca\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.177181 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.177116    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-oauth-serving-cert\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.177181 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.177154    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-trusted-ca-bundle\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.177181 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.177178    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00e32261-75cf-4304-8605-738a3467ac45-console-serving-cert\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.177860 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.177831    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-oauth-serving-cert\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.177996 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.177882    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-console-config\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.178074 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.178052    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-service-ca\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.178197 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.178179    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-trusted-ca-bundle\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.179523 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.179496    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00e32261-75cf-4304-8605-738a3467ac45-console-oauth-config\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.179625 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.179606    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00e32261-75cf-4304-8605-738a3467ac45-console-serving-cert\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.185197 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.185175    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph8q7\" (UniqueName: \"kubernetes.io/projected/00e32261-75cf-4304-8605-738a3467ac45-kube-api-access-ph8q7\") pod \"console-76d74ffb9-hx6xl\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") " pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.280073 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.280051    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:24.405966 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.405889    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76d74ffb9-hx6xl"]
Jul 23 13:43:24.408546 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:43:24.408521    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e32261_75cf_4304_8605_738a3467ac45.slice/crio-efb7811b16fee8b8de7a32fd494404414477796f8b97fe05fd140ab782f668e5 WatchSource:0}: Error finding container efb7811b16fee8b8de7a32fd494404414477796f8b97fe05fd140ab782f668e5: Status 404 returned error can't find the container with id efb7811b16fee8b8de7a32fd494404414477796f8b97fe05fd140ab782f668e5
Jul 23 13:43:24.810959 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.810923    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d74ffb9-hx6xl" event={"ID":"00e32261-75cf-4304-8605-738a3467ac45","Type":"ContainerStarted","Data":"7bf161955162180e3522fded0481c76b3fff0e4365fdce65fc14b8f79deba887"}
Jul 23 13:43:24.811142 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.810985    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d74ffb9-hx6xl" event={"ID":"00e32261-75cf-4304-8605-738a3467ac45","Type":"ContainerStarted","Data":"efb7811b16fee8b8de7a32fd494404414477796f8b97fe05fd140ab782f668e5"}
Jul 23 13:43:24.830404 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:24.830357    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76d74ffb9-hx6xl" podStartSLOduration=1.83034213 podStartE2EDuration="1.83034213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:43:24.829764049 +0000 UTC m=+153.290855091" watchObservedRunningTime="2026-07-23 13:43:24.83034213 +0000 UTC m=+153.291433240"
Jul 23 13:43:34.280247 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:34.280217    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:34.280726 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:34.280276    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:34.284965 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:34.284938    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:34.852241 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:34.852213    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:43:34.906272 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:34.906245    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54d77cc564-v9wk6"]
Jul 23 13:43:59.926337 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:43:59.926289    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54d77cc564-v9wk6" podUID="5b050fd9-b8b1-471f-b826-077cc0650a46" containerName="console" containerID="cri-o://77f417e02b618ad51d62d746a37f8dad55802c1ec5524b5b39133f56c677d296" gracePeriod=15
Jul 23 13:44:00.170564 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.170542    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:44:00.249999 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.249974    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tddw2\" (UniqueName: \"kubernetes.io/projected/5b050fd9-b8b1-471f-b826-077cc0650a46-kube-api-access-tddw2\") pod \"5b050fd9-b8b1-471f-b826-077cc0650a46\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") "
Jul 23 13:44:00.250150 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.250008    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-service-ca\") pod \"5b050fd9-b8b1-471f-b826-077cc0650a46\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") "
Jul 23 13:44:00.250150 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.250029    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b050fd9-b8b1-471f-b826-077cc0650a46-console-oauth-config\") pod \"5b050fd9-b8b1-471f-b826-077cc0650a46\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") "
Jul 23 13:44:00.250150 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.250069    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-trusted-ca-bundle\") pod \"5b050fd9-b8b1-471f-b826-077cc0650a46\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") "
Jul 23 13:44:00.250150 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.250090    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-oauth-serving-cert\") pod \"5b050fd9-b8b1-471f-b826-077cc0650a46\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") "
Jul 23 13:44:00.250150 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.250116    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b050fd9-b8b1-471f-b826-077cc0650a46-console-serving-cert\") pod \"5b050fd9-b8b1-471f-b826-077cc0650a46\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") "
Jul 23 13:44:00.250401 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.250156    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-console-config\") pod \"5b050fd9-b8b1-471f-b826-077cc0650a46\" (UID: \"5b050fd9-b8b1-471f-b826-077cc0650a46\") "
Jul 23 13:44:00.250456 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.250436    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-service-ca" (OuterVolumeSpecName: "service-ca") pod "5b050fd9-b8b1-471f-b826-077cc0650a46" (UID: "5b050fd9-b8b1-471f-b826-077cc0650a46"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:44:00.250697 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.250527    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5b050fd9-b8b1-471f-b826-077cc0650a46" (UID: "5b050fd9-b8b1-471f-b826-077cc0650a46"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:44:00.250697 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.250535    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5b050fd9-b8b1-471f-b826-077cc0650a46" (UID: "5b050fd9-b8b1-471f-b826-077cc0650a46"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:44:00.250887 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.250753    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-console-config" (OuterVolumeSpecName: "console-config") pod "5b050fd9-b8b1-471f-b826-077cc0650a46" (UID: "5b050fd9-b8b1-471f-b826-077cc0650a46"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:44:00.252266 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.252238    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b050fd9-b8b1-471f-b826-077cc0650a46-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5b050fd9-b8b1-471f-b826-077cc0650a46" (UID: "5b050fd9-b8b1-471f-b826-077cc0650a46"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:44:00.252368 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.252297    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b050fd9-b8b1-471f-b826-077cc0650a46-kube-api-access-tddw2" (OuterVolumeSpecName: "kube-api-access-tddw2") pod "5b050fd9-b8b1-471f-b826-077cc0650a46" (UID: "5b050fd9-b8b1-471f-b826-077cc0650a46"). InnerVolumeSpecName "kube-api-access-tddw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 13:44:00.252368 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.252303    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b050fd9-b8b1-471f-b826-077cc0650a46-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5b050fd9-b8b1-471f-b826-077cc0650a46" (UID: "5b050fd9-b8b1-471f-b826-077cc0650a46"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:44:00.350786 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.350751    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tddw2\" (UniqueName: \"kubernetes.io/projected/5b050fd9-b8b1-471f-b826-077cc0650a46-kube-api-access-tddw2\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:44:00.350786 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.350780    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-service-ca\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:44:00.350786 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.350790    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b050fd9-b8b1-471f-b826-077cc0650a46-console-oauth-config\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:44:00.351013 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.350798    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-trusted-ca-bundle\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:44:00.351013 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.350807    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-oauth-serving-cert\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:44:00.351013 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.350816    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b050fd9-b8b1-471f-b826-077cc0650a46-console-serving-cert\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:44:00.351013 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.350825    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b050fd9-b8b1-471f-b826-077cc0650a46-console-config\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:44:00.941379 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.941338    2568 generic.go:356] "Generic (PLEG): container finished" podID="5b050fd9-b8b1-471f-b826-077cc0650a46" containerID="77f417e02b618ad51d62d746a37f8dad55802c1ec5524b5b39133f56c677d296" exitCode=0
Jul 23 13:44:00.941788 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.941389    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d77cc564-v9wk6" event={"ID":"5b050fd9-b8b1-471f-b826-077cc0650a46","Type":"ContainerDied","Data":"77f417e02b618ad51d62d746a37f8dad55802c1ec5524b5b39133f56c677d296"}
Jul 23 13:44:00.941788 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.941402    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d77cc564-v9wk6"
Jul 23 13:44:00.941788 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.941476    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d77cc564-v9wk6" event={"ID":"5b050fd9-b8b1-471f-b826-077cc0650a46","Type":"ContainerDied","Data":"29f9bc8dd6f39be13240562c4a70bcd453b5ad4ea14e8d2a5d298a936defbc9d"}
Jul 23 13:44:00.941788 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.941505    2568 scope.go:117] "RemoveContainer" containerID="77f417e02b618ad51d62d746a37f8dad55802c1ec5524b5b39133f56c677d296"
Jul 23 13:44:00.950267 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.950249    2568 scope.go:117] "RemoveContainer" containerID="77f417e02b618ad51d62d746a37f8dad55802c1ec5524b5b39133f56c677d296"
Jul 23 13:44:00.950552 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:44:00.950534    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f417e02b618ad51d62d746a37f8dad55802c1ec5524b5b39133f56c677d296\": container with ID starting with 77f417e02b618ad51d62d746a37f8dad55802c1ec5524b5b39133f56c677d296 not found: ID does not exist" containerID="77f417e02b618ad51d62d746a37f8dad55802c1ec5524b5b39133f56c677d296"
Jul 23 13:44:00.950599 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.950561    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f417e02b618ad51d62d746a37f8dad55802c1ec5524b5b39133f56c677d296"} err="failed to get container status \"77f417e02b618ad51d62d746a37f8dad55802c1ec5524b5b39133f56c677d296\": rpc error: code = NotFound desc = could not find container \"77f417e02b618ad51d62d746a37f8dad55802c1ec5524b5b39133f56c677d296\": container with ID starting with 77f417e02b618ad51d62d746a37f8dad55802c1ec5524b5b39133f56c677d296 not found: ID does not exist"
Jul 23 13:44:00.965698 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.965673    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54d77cc564-v9wk6"]
Jul 23 13:44:00.972846 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:00.972825    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54d77cc564-v9wk6"]
Jul 23 13:44:02.149573 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:02.149544    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b050fd9-b8b1-471f-b826-077cc0650a46" path="/var/lib/kubelet/pods/5b050fd9-b8b1-471f-b826-077cc0650a46/volumes"
Jul 23 13:44:37.799057 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.799024    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-87d9c6bfd-5d9g2"]
Jul 23 13:44:37.799556 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.799323    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b050fd9-b8b1-471f-b826-077cc0650a46" containerName="console"
Jul 23 13:44:37.799556 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.799334    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b050fd9-b8b1-471f-b826-077cc0650a46" containerName="console"
Jul 23 13:44:37.799556 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.799394    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b050fd9-b8b1-471f-b826-077cc0650a46" containerName="console"
Jul 23 13:44:37.802324 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.802307    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.821515 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.821489    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87d9c6bfd-5d9g2"]
Jul 23 13:44:37.846461 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.846440    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-oauth-config\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.846570 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.846472    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-service-ca\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.846570 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.846495    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-serving-cert\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.846570 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.846525    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-trusted-ca-bundle\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.846673 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.846596    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-config\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.846673 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.846621    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td8lq\" (UniqueName: \"kubernetes.io/projected/f0561c6a-fb2d-4479-ae8d-0231a913f826-kube-api-access-td8lq\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.846673 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.846659    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-oauth-serving-cert\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.948017 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.947977    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-config\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.948017 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.948016    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td8lq\" (UniqueName: \"kubernetes.io/projected/f0561c6a-fb2d-4479-ae8d-0231a913f826-kube-api-access-td8lq\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.948229 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.948062    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-oauth-serving-cert\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.948229 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.948109    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-oauth-config\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.948229 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.948147    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-service-ca\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.948371 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.948274    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-serving-cert\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.948371 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.948337    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-trusted-ca-bundle\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.948901 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.948878    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-config\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.949031 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.948878    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-oauth-serving-cert\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.949031 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.948918    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-service-ca\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.949138 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.949081    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-trusted-ca-bundle\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.950633 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.950613    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-serving-cert\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.950708 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.950691    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-oauth-config\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:37.957475 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:37.957450    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td8lq\" (UniqueName: \"kubernetes.io/projected/f0561c6a-fb2d-4479-ae8d-0231a913f826-kube-api-access-td8lq\") pod \"console-87d9c6bfd-5d9g2\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") " pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:38.110994 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:38.110920    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:38.235023 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:38.234995    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87d9c6bfd-5d9g2"]
Jul 23 13:44:38.237491 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:44:38.237463    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0561c6a_fb2d_4479_ae8d_0231a913f826.slice/crio-4e745192801420efbe020a159b90ef1af4b4c66fc4cae0725e79737476a550f2 WatchSource:0}: Error finding container 4e745192801420efbe020a159b90ef1af4b4c66fc4cae0725e79737476a550f2: Status 404 returned error can't find the container with id 4e745192801420efbe020a159b90ef1af4b4c66fc4cae0725e79737476a550f2
Jul 23 13:44:39.082597 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:39.082562    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87d9c6bfd-5d9g2" event={"ID":"f0561c6a-fb2d-4479-ae8d-0231a913f826","Type":"ContainerStarted","Data":"8c035fcd2b52669314e76dfb40cb686c5da21705e4ef9b67de71d2fafc99c785"}
Jul 23 13:44:39.083066 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:39.082640    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87d9c6bfd-5d9g2" event={"ID":"f0561c6a-fb2d-4479-ae8d-0231a913f826","Type":"ContainerStarted","Data":"4e745192801420efbe020a159b90ef1af4b4c66fc4cae0725e79737476a550f2"}
Jul 23 13:44:39.100332 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:39.100287    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-87d9c6bfd-5d9g2" podStartSLOduration=2.100272686 podStartE2EDuration="2.100272686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:44:39.099941854 +0000 UTC m=+227.561032895" watchObservedRunningTime="2026-07-23 13:44:39.100272686 +0000 UTC m=+227.561363728"
Jul 23 13:44:48.111175 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:48.111138    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:48.111571 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:48.111238    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:48.116277 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:48.116253    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:48.120775 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:48.120757    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:44:48.200654 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:44:48.200624    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76d74ffb9-hx6xl"]
Jul 23 13:45:12.621916 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.621885    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"]
Jul 23 13:45:12.625308 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.625285    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.629542 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.629524    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 23 13:45:12.629809 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.629789    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 23 13:45:12.629923 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.629835    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 23 13:45:12.630943 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.630920    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 23 13:45:12.631037 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.630923    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 23 13:45:12.631037 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.630962    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 23 13:45:12.631037 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.630991    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 23 13:45:12.644452 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.644425    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"]
Jul 23 13:45:12.725865 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.725813    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d5fd9935-45ae-44cc-a2b3-1718490c3e56-ca\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.725865 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.725841    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d5fd9935-45ae-44cc-a2b3-1718490c3e56-hub\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.726049 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.725907    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fd9935-45ae-44cc-a2b3-1718490c3e56-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.726049 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.725943    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5fd9935-45ae-44cc-a2b3-1718490c3e56-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.726049 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.725994    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d5fd9935-45ae-44cc-a2b3-1718490c3e56-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.726049 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.726038    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbjq5\" (UniqueName: \"kubernetes.io/projected/d5fd9935-45ae-44cc-a2b3-1718490c3e56-kube-api-access-gbjq5\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.785023 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.784973    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5649554f94-8xf5p"]
Jul 23 13:45:12.789210 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.789185    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5649554f94-8xf5p"
Jul 23 13:45:12.792300 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.792274    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 23 13:45:12.792418 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.792313    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-g255v\""
Jul 23 13:45:12.799751 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.799729    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5649554f94-8xf5p"]
Jul 23 13:45:12.826637 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.826614    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9ba8a0d-5890-4701-ae8e-4e85629f070a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5649554f94-8xf5p\" (UID: \"e9ba8a0d-5890-4701-ae8e-4e85629f070a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5649554f94-8xf5p"
Jul 23 13:45:12.826747 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.826673    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5fd9935-45ae-44cc-a2b3-1718490c3e56-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.826747 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.826714    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d5fd9935-45ae-44cc-a2b3-1718490c3e56-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.826894 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.826754    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvgv4\" (UniqueName: \"kubernetes.io/projected/e9ba8a0d-5890-4701-ae8e-4e85629f070a-kube-api-access-gvgv4\") pod \"managed-serviceaccount-addon-agent-5649554f94-8xf5p\" (UID: \"e9ba8a0d-5890-4701-ae8e-4e85629f070a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5649554f94-8xf5p"
Jul 23 13:45:12.826894 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.826787    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbjq5\" (UniqueName: \"kubernetes.io/projected/d5fd9935-45ae-44cc-a2b3-1718490c3e56-kube-api-access-gbjq5\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.826894 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.826868    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d5fd9935-45ae-44cc-a2b3-1718490c3e56-ca\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.827051 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.826892    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d5fd9935-45ae-44cc-a2b3-1718490c3e56-hub\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.827051 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.826933    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fd9935-45ae-44cc-a2b3-1718490c3e56-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.827638 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.827610    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fd9935-45ae-44cc-a2b3-1718490c3e56-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.829258 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.829234    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d5fd9935-45ae-44cc-a2b3-1718490c3e56-ca\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.829340 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.829296    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d5fd9935-45ae-44cc-a2b3-1718490c3e56-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.829382 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.829333    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d5fd9935-45ae-44cc-a2b3-1718490c3e56-hub\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.829507 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.829485    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5fd9935-45ae-44cc-a2b3-1718490c3e56-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.835479 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.835454    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbjq5\" (UniqueName: \"kubernetes.io/projected/d5fd9935-45ae-44cc-a2b3-1718490c3e56-kube-api-access-gbjq5\") pod \"cluster-proxy-proxy-agent-6b854d9fb9-kk48d\" (UID: \"d5fd9935-45ae-44cc-a2b3-1718490c3e56\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:12.928025 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.927954    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvgv4\" (UniqueName: \"kubernetes.io/projected/e9ba8a0d-5890-4701-ae8e-4e85629f070a-kube-api-access-gvgv4\") pod \"managed-serviceaccount-addon-agent-5649554f94-8xf5p\" (UID: \"e9ba8a0d-5890-4701-ae8e-4e85629f070a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5649554f94-8xf5p"
Jul 23 13:45:12.928144 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.928051    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9ba8a0d-5890-4701-ae8e-4e85629f070a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5649554f94-8xf5p\" (UID: \"e9ba8a0d-5890-4701-ae8e-4e85629f070a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5649554f94-8xf5p"
Jul 23 13:45:12.930366 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.930338    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9ba8a0d-5890-4701-ae8e-4e85629f070a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5649554f94-8xf5p\" (UID: \"e9ba8a0d-5890-4701-ae8e-4e85629f070a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5649554f94-8xf5p"
Jul 23 13:45:12.937095 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.937069    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvgv4\" (UniqueName: \"kubernetes.io/projected/e9ba8a0d-5890-4701-ae8e-4e85629f070a-kube-api-access-gvgv4\") pod \"managed-serviceaccount-addon-agent-5649554f94-8xf5p\" (UID: \"e9ba8a0d-5890-4701-ae8e-4e85629f070a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5649554f94-8xf5p"
Jul 23 13:45:12.946755 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:12.946738    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"
Jul 23 13:45:13.079018 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.078926    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d"]
Jul 23 13:45:13.081584 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:45:13.081560    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5fd9935_45ae_44cc_a2b3_1718490c3e56.slice/crio-10a90a041b347e93bff6f3081a356726160844ea8a963080930ee01ffdeedce8 WatchSource:0}: Error finding container 10a90a041b347e93bff6f3081a356726160844ea8a963080930ee01ffdeedce8: Status 404 returned error can't find the container with id 10a90a041b347e93bff6f3081a356726160844ea8a963080930ee01ffdeedce8
Jul 23 13:45:13.097386 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.097367    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5649554f94-8xf5p"
Jul 23 13:45:13.206021 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.205950    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d" event={"ID":"d5fd9935-45ae-44cc-a2b3-1718490c3e56","Type":"ContainerStarted","Data":"10a90a041b347e93bff6f3081a356726160844ea8a963080930ee01ffdeedce8"}
Jul 23 13:45:13.221486 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.221441    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76d74ffb9-hx6xl" podUID="00e32261-75cf-4304-8605-738a3467ac45" containerName="console" containerID="cri-o://7bf161955162180e3522fded0481c76b3fff0e4365fdce65fc14b8f79deba887" gracePeriod=15
Jul 23 13:45:13.223209 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.223132    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5649554f94-8xf5p"]
Jul 23 13:45:13.225217 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:45:13.225182    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9ba8a0d_5890_4701_ae8e_4e85629f070a.slice/crio-b4f02124055daa040e1b09ceaa6b18122b363fde1e3c2cb142c6033ca1f5aa86 WatchSource:0}: Error finding container b4f02124055daa040e1b09ceaa6b18122b363fde1e3c2cb142c6033ca1f5aa86: Status 404 returned error can't find the container with id b4f02124055daa040e1b09ceaa6b18122b363fde1e3c2cb142c6033ca1f5aa86
Jul 23 13:45:13.449900 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.449872    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:45:13.534670 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.534644    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00e32261-75cf-4304-8605-738a3467ac45-console-serving-cert\") pod \"00e32261-75cf-4304-8605-738a3467ac45\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") "
Jul 23 13:45:13.534839 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.534691    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00e32261-75cf-4304-8605-738a3467ac45-console-oauth-config\") pod \"00e32261-75cf-4304-8605-738a3467ac45\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") "
Jul 23 13:45:13.534839 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.534728    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-trusted-ca-bundle\") pod \"00e32261-75cf-4304-8605-738a3467ac45\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") "
Jul 23 13:45:13.534839 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.534767    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-service-ca\") pod \"00e32261-75cf-4304-8605-738a3467ac45\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") "
Jul 23 13:45:13.534839 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.534793    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph8q7\" (UniqueName: \"kubernetes.io/projected/00e32261-75cf-4304-8605-738a3467ac45-kube-api-access-ph8q7\") pod \"00e32261-75cf-4304-8605-738a3467ac45\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") "
Jul 23 13:45:13.535079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.534882    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-console-config\") pod \"00e32261-75cf-4304-8605-738a3467ac45\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") "
Jul 23 13:45:13.535079 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.534927    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-oauth-serving-cert\") pod \"00e32261-75cf-4304-8605-738a3467ac45\" (UID: \"00e32261-75cf-4304-8605-738a3467ac45\") "
Jul 23 13:45:13.535295 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.535266    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "00e32261-75cf-4304-8605-738a3467ac45" (UID: "00e32261-75cf-4304-8605-738a3467ac45"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:45:13.535422 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.535369    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "00e32261-75cf-4304-8605-738a3467ac45" (UID: "00e32261-75cf-4304-8605-738a3467ac45"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:45:13.535543 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.535484    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-service-ca" (OuterVolumeSpecName: "service-ca") pod "00e32261-75cf-4304-8605-738a3467ac45" (UID: "00e32261-75cf-4304-8605-738a3467ac45"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:45:13.535543 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.535505    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-console-config" (OuterVolumeSpecName: "console-config") pod "00e32261-75cf-4304-8605-738a3467ac45" (UID: "00e32261-75cf-4304-8605-738a3467ac45"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:45:13.536794 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.536775    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00e32261-75cf-4304-8605-738a3467ac45-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "00e32261-75cf-4304-8605-738a3467ac45" (UID: "00e32261-75cf-4304-8605-738a3467ac45"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:45:13.537186 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.537166    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00e32261-75cf-4304-8605-738a3467ac45-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "00e32261-75cf-4304-8605-738a3467ac45" (UID: "00e32261-75cf-4304-8605-738a3467ac45"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:45:13.537254 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.537235    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e32261-75cf-4304-8605-738a3467ac45-kube-api-access-ph8q7" (OuterVolumeSpecName: "kube-api-access-ph8q7") pod "00e32261-75cf-4304-8605-738a3467ac45" (UID: "00e32261-75cf-4304-8605-738a3467ac45"). InnerVolumeSpecName "kube-api-access-ph8q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 13:45:13.636218 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.636188    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-oauth-serving-cert\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:45:13.636218 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.636211    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00e32261-75cf-4304-8605-738a3467ac45-console-serving-cert\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:45:13.636218 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.636221    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00e32261-75cf-4304-8605-738a3467ac45-console-oauth-config\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:45:13.636610 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.636231    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-trusted-ca-bundle\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:45:13.636610 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.636240    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-service-ca\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:45:13.636610 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.636250    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph8q7\" (UniqueName: \"kubernetes.io/projected/00e32261-75cf-4304-8605-738a3467ac45-kube-api-access-ph8q7\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:45:13.636610 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:13.636259    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00e32261-75cf-4304-8605-738a3467ac45-console-config\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:45:14.214429 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:14.214068    2568 generic.go:356] "Generic (PLEG): container finished" podID="00e32261-75cf-4304-8605-738a3467ac45" containerID="7bf161955162180e3522fded0481c76b3fff0e4365fdce65fc14b8f79deba887" exitCode=0
Jul 23 13:45:14.214429 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:14.214181    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d74ffb9-hx6xl"
Jul 23 13:45:14.214429 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:14.214216    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d74ffb9-hx6xl" event={"ID":"00e32261-75cf-4304-8605-738a3467ac45","Type":"ContainerDied","Data":"7bf161955162180e3522fded0481c76b3fff0e4365fdce65fc14b8f79deba887"}
Jul 23 13:45:14.214429 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:14.214263    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d74ffb9-hx6xl" event={"ID":"00e32261-75cf-4304-8605-738a3467ac45","Type":"ContainerDied","Data":"efb7811b16fee8b8de7a32fd494404414477796f8b97fe05fd140ab782f668e5"}
Jul 23 13:45:14.214429 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:14.214293    2568 scope.go:117] "RemoveContainer" containerID="7bf161955162180e3522fded0481c76b3fff0e4365fdce65fc14b8f79deba887"
Jul 23 13:45:14.217021 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:14.216953    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5649554f94-8xf5p" event={"ID":"e9ba8a0d-5890-4701-ae8e-4e85629f070a","Type":"ContainerStarted","Data":"b4f02124055daa040e1b09ceaa6b18122b363fde1e3c2cb142c6033ca1f5aa86"}
Jul 23 13:45:14.232758 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:14.232737    2568 scope.go:117] "RemoveContainer" containerID="7bf161955162180e3522fded0481c76b3fff0e4365fdce65fc14b8f79deba887"
Jul 23 13:45:14.234174 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:45:14.234093    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bf161955162180e3522fded0481c76b3fff0e4365fdce65fc14b8f79deba887\": container with ID starting with 7bf161955162180e3522fded0481c76b3fff0e4365fdce65fc14b8f79deba887 not found: ID does not exist" containerID="7bf161955162180e3522fded0481c76b3fff0e4365fdce65fc14b8f79deba887"
Jul 23 13:45:14.234174 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:14.234131    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf161955162180e3522fded0481c76b3fff0e4365fdce65fc14b8f79deba887"} err="failed to get container status \"7bf161955162180e3522fded0481c76b3fff0e4365fdce65fc14b8f79deba887\": rpc error: code = NotFound desc = could not find container \"7bf161955162180e3522fded0481c76b3fff0e4365fdce65fc14b8f79deba887\": container with ID starting with 7bf161955162180e3522fded0481c76b3fff0e4365fdce65fc14b8f79deba887 not found: ID does not exist"
Jul 23 13:45:14.234333 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:14.234238    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76d74ffb9-hx6xl"]
Jul 23 13:45:14.238992 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:14.238946    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76d74ffb9-hx6xl"]
Jul 23 13:45:16.151731 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:16.151696    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00e32261-75cf-4304-8605-738a3467ac45" path="/var/lib/kubelet/pods/00e32261-75cf-4304-8605-738a3467ac45/volumes"
Jul 23 13:45:18.238054 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:18.238012    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d" event={"ID":"d5fd9935-45ae-44cc-a2b3-1718490c3e56","Type":"ContainerStarted","Data":"e57a59302dcb3ca43372ea5f076495914bbaa099f48ddf9f4d3ce84d62a7463a"}
Jul 23 13:45:18.238609 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:18.238063    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d" event={"ID":"d5fd9935-45ae-44cc-a2b3-1718490c3e56","Type":"ContainerStarted","Data":"a8338a75d2da1819cf2c2befa6f88ab2657190fd96736fc9a094f584e75ad0cd"}
Jul 23 13:45:18.238609 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:18.238083    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d" event={"ID":"d5fd9935-45ae-44cc-a2b3-1718490c3e56","Type":"ContainerStarted","Data":"c186f75a1451308fa52a8f8cb4fc6e645262554b4875ee55d32ffb41c3a21e38"}
Jul 23 13:45:18.239414 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:18.239389    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5649554f94-8xf5p" event={"ID":"e9ba8a0d-5890-4701-ae8e-4e85629f070a","Type":"ContainerStarted","Data":"6edd44d8c75c8c3e4c736d2ca73000d8d09b0b76ecd713a627b3e6cdfe909e4c"}
Jul 23 13:45:18.258885 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:18.258823    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b854d9fb9-kk48d" podStartSLOduration=2.124657675 podStartE2EDuration="6.258811192s" totalImagesPullingTime="4.134153517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:45:18.257666417 +0000 UTC m=+266.718757460" watchObservedRunningTime="2026-07-23 13:45:18.258811192 +0000 UTC m=+266.719902234"
Jul 23 13:45:18.272360 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:18.272312    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5649554f94-8xf5p" podStartSLOduration=2.293532711 podStartE2EDuration="6.272292967s" totalImagesPullingTime="3.978760256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:45:18.271985609 +0000 UTC m=+266.733076659" watchObservedRunningTime="2026-07-23 13:45:18.272292967 +0000 UTC m=+266.733384010"
Jul 23 13:45:51.997842 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:45:51.997817    2568 kubelet.go:1648] "Image garbage collection succeeded"
Jul 23 13:46:43.931129 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:43.931040    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"]
Jul 23 13:46:43.933098 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:43.931498    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e32261-75cf-4304-8605-738a3467ac45" containerName="console"
Jul 23 13:46:43.933098 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:43.931516    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e32261-75cf-4304-8605-738a3467ac45" containerName="console"
Jul 23 13:46:43.933098 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:43.931579    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e32261-75cf-4304-8605-738a3467ac45" containerName="console"
Jul 23 13:46:43.933760 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:43.933740    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"
Jul 23 13:46:43.939383 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:43.939362    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-b2bjn\""
Jul 23 13:46:43.939504 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:43.939362    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 23 13:46:43.939504 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:43.939364    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 23 13:46:43.939504 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:43.939483    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 23 13:46:43.939504 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:43.939485    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 23 13:46:43.939697 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:43.939439    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 23 13:46:43.948317 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:43.948292    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"]
Jul 23 13:46:44.036790 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:44.036759    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b88a3fe2-f5f7-4da0-bb14-6c11561cf340-metrics-cert\") pod \"lws-controller-manager-6d8d9c69b6-j5swr\" (UID: \"b88a3fe2-f5f7-4da0-bb14-6c11561cf340\") " pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"
Jul 23 13:46:44.036790 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:44.036792    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b88a3fe2-f5f7-4da0-bb14-6c11561cf340-manager-config\") pod \"lws-controller-manager-6d8d9c69b6-j5swr\" (UID: \"b88a3fe2-f5f7-4da0-bb14-6c11561cf340\") " pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"
Jul 23 13:46:44.037013 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:44.036817    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b88a3fe2-f5f7-4da0-bb14-6c11561cf340-cert\") pod \"lws-controller-manager-6d8d9c69b6-j5swr\" (UID: \"b88a3fe2-f5f7-4da0-bb14-6c11561cf340\") " pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"
Jul 23 13:46:44.037013 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:44.036872    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9bg4\" (UniqueName: \"kubernetes.io/projected/b88a3fe2-f5f7-4da0-bb14-6c11561cf340-kube-api-access-x9bg4\") pod \"lws-controller-manager-6d8d9c69b6-j5swr\" (UID: \"b88a3fe2-f5f7-4da0-bb14-6c11561cf340\") " pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"
Jul 23 13:46:44.137602 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:44.137568    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b88a3fe2-f5f7-4da0-bb14-6c11561cf340-metrics-cert\") pod \"lws-controller-manager-6d8d9c69b6-j5swr\" (UID: \"b88a3fe2-f5f7-4da0-bb14-6c11561cf340\") " pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"
Jul 23 13:46:44.137602 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:44.137604    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b88a3fe2-f5f7-4da0-bb14-6c11561cf340-manager-config\") pod \"lws-controller-manager-6d8d9c69b6-j5swr\" (UID: \"b88a3fe2-f5f7-4da0-bb14-6c11561cf340\") " pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"
Jul 23 13:46:44.137807 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:44.137636    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b88a3fe2-f5f7-4da0-bb14-6c11561cf340-cert\") pod \"lws-controller-manager-6d8d9c69b6-j5swr\" (UID: \"b88a3fe2-f5f7-4da0-bb14-6c11561cf340\") " pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"
Jul 23 13:46:44.137807 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:44.137731    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9bg4\" (UniqueName: \"kubernetes.io/projected/b88a3fe2-f5f7-4da0-bb14-6c11561cf340-kube-api-access-x9bg4\") pod \"lws-controller-manager-6d8d9c69b6-j5swr\" (UID: \"b88a3fe2-f5f7-4da0-bb14-6c11561cf340\") " pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"
Jul 23 13:46:44.138497 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:44.138467    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b88a3fe2-f5f7-4da0-bb14-6c11561cf340-manager-config\") pod \"lws-controller-manager-6d8d9c69b6-j5swr\" (UID: \"b88a3fe2-f5f7-4da0-bb14-6c11561cf340\") " pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"
Jul 23 13:46:44.140244 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:44.140214    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b88a3fe2-f5f7-4da0-bb14-6c11561cf340-cert\") pod \"lws-controller-manager-6d8d9c69b6-j5swr\" (UID: \"b88a3fe2-f5f7-4da0-bb14-6c11561cf340\") " pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"
Jul 23 13:46:44.140244 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:44.140239    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b88a3fe2-f5f7-4da0-bb14-6c11561cf340-metrics-cert\") pod \"lws-controller-manager-6d8d9c69b6-j5swr\" (UID: \"b88a3fe2-f5f7-4da0-bb14-6c11561cf340\") " pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"
Jul 23 13:46:44.154157 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:44.154135    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9bg4\" (UniqueName: \"kubernetes.io/projected/b88a3fe2-f5f7-4da0-bb14-6c11561cf340-kube-api-access-x9bg4\") pod \"lws-controller-manager-6d8d9c69b6-j5swr\" (UID: \"b88a3fe2-f5f7-4da0-bb14-6c11561cf340\") " pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"
Jul 23 13:46:44.242416 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:44.242389    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"
Jul 23 13:46:44.377765 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:44.377735    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"]
Jul 23 13:46:44.380037 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:46:44.380010    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88a3fe2_f5f7_4da0_bb14_6c11561cf340.slice/crio-bb737986a3d3a8fc5e197f5172d2d887e4923194642205d615317d179a26407d WatchSource:0}: Error finding container bb737986a3d3a8fc5e197f5172d2d887e4923194642205d615317d179a26407d: Status 404 returned error can't find the container with id bb737986a3d3a8fc5e197f5172d2d887e4923194642205d615317d179a26407d
Jul 23 13:46:44.381806 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:44.381788    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 13:46:44.554738 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:44.554670    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr" event={"ID":"b88a3fe2-f5f7-4da0-bb14-6c11561cf340","Type":"ContainerStarted","Data":"bb737986a3d3a8fc5e197f5172d2d887e4923194642205d615317d179a26407d"}
Jul 23 13:46:47.520902 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.520847    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"]
Jul 23 13:46:47.523360 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.523344    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.531083 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.531060    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 23 13:46:47.531280 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.531267    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 23 13:46:47.532711 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.532691    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 23 13:46:47.535244 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.535228    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 23 13:46:47.538367 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.538347    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4qrml\""
Jul 23 13:46:47.566963 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.566940    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"]
Jul 23 13:46:47.570571 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.570543    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr" event={"ID":"b88a3fe2-f5f7-4da0-bb14-6c11561cf340","Type":"ContainerStarted","Data":"b167b3df1af326c979314cd06e63ea6dc7f191180f8cc4b27c5c619b9305814c"}
Jul 23 13:46:47.570665 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.570600    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"
Jul 23 13:46:47.612128 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.612078    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr" podStartSLOduration=2.347716599 podStartE2EDuration="4.612063033s" totalImagesPullingTime="2.264346434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:46:47.610698452 +0000 UTC m=+356.071789494" watchObservedRunningTime="2026-07-23 13:46:47.612063033 +0000 UTC m=+356.073154134"
Jul 23 13:46:47.667179 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.667151    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78a34ac0-ed10-48a3-ab9e-deb9e54c8989-manifests\") pod \"opendatahub-operator-controller-manager-7c8b88fb69-4snf8\" (UID: \"78a34ac0-ed10-48a3-ab9e-deb9e54c8989\") " pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.667280 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.667187    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvfhw\" (UniqueName: \"kubernetes.io/projected/78a34ac0-ed10-48a3-ab9e-deb9e54c8989-kube-api-access-jvfhw\") pod \"opendatahub-operator-controller-manager-7c8b88fb69-4snf8\" (UID: \"78a34ac0-ed10-48a3-ab9e-deb9e54c8989\") " pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.667280 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.667212    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78a34ac0-ed10-48a3-ab9e-deb9e54c8989-tmp\") pod \"opendatahub-operator-controller-manager-7c8b88fb69-4snf8\" (UID: \"78a34ac0-ed10-48a3-ab9e-deb9e54c8989\") " pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.667280 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.667233    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78a34ac0-ed10-48a3-ab9e-deb9e54c8989-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c8b88fb69-4snf8\" (UID: \"78a34ac0-ed10-48a3-ab9e-deb9e54c8989\") " pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.667445 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.667337    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/78a34ac0-ed10-48a3-ab9e-deb9e54c8989-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c8b88fb69-4snf8\" (UID: \"78a34ac0-ed10-48a3-ab9e-deb9e54c8989\") " pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.768408 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.768379    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78a34ac0-ed10-48a3-ab9e-deb9e54c8989-tmp\") pod \"opendatahub-operator-controller-manager-7c8b88fb69-4snf8\" (UID: \"78a34ac0-ed10-48a3-ab9e-deb9e54c8989\") " pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.768532 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.768418    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78a34ac0-ed10-48a3-ab9e-deb9e54c8989-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c8b88fb69-4snf8\" (UID: \"78a34ac0-ed10-48a3-ab9e-deb9e54c8989\") " pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.768532 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.768453    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/78a34ac0-ed10-48a3-ab9e-deb9e54c8989-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c8b88fb69-4snf8\" (UID: \"78a34ac0-ed10-48a3-ab9e-deb9e54c8989\") " pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.768532 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.768490    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78a34ac0-ed10-48a3-ab9e-deb9e54c8989-manifests\") pod \"opendatahub-operator-controller-manager-7c8b88fb69-4snf8\" (UID: \"78a34ac0-ed10-48a3-ab9e-deb9e54c8989\") " pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.768532 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.768511    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvfhw\" (UniqueName: \"kubernetes.io/projected/78a34ac0-ed10-48a3-ab9e-deb9e54c8989-kube-api-access-jvfhw\") pod \"opendatahub-operator-controller-manager-7c8b88fb69-4snf8\" (UID: \"78a34ac0-ed10-48a3-ab9e-deb9e54c8989\") " pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.768913 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.768885    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78a34ac0-ed10-48a3-ab9e-deb9e54c8989-manifests\") pod \"opendatahub-operator-controller-manager-7c8b88fb69-4snf8\" (UID: \"78a34ac0-ed10-48a3-ab9e-deb9e54c8989\") " pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.770787 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.770763    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78a34ac0-ed10-48a3-ab9e-deb9e54c8989-tmp\") pod \"opendatahub-operator-controller-manager-7c8b88fb69-4snf8\" (UID: \"78a34ac0-ed10-48a3-ab9e-deb9e54c8989\") " pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.771043 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.771005    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78a34ac0-ed10-48a3-ab9e-deb9e54c8989-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c8b88fb69-4snf8\" (UID: \"78a34ac0-ed10-48a3-ab9e-deb9e54c8989\") " pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.771043 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.771021    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/78a34ac0-ed10-48a3-ab9e-deb9e54c8989-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c8b88fb69-4snf8\" (UID: \"78a34ac0-ed10-48a3-ab9e-deb9e54c8989\") " pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.776542 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.776516    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvfhw\" (UniqueName: \"kubernetes.io/projected/78a34ac0-ed10-48a3-ab9e-deb9e54c8989-kube-api-access-jvfhw\") pod \"opendatahub-operator-controller-manager-7c8b88fb69-4snf8\" (UID: \"78a34ac0-ed10-48a3-ab9e-deb9e54c8989\") " pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.833214 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.833192    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:47.969726 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:47.969695    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"]
Jul 23 13:46:47.974321 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:46:47.974294    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a34ac0_ed10_48a3_ab9e_deb9e54c8989.slice/crio-4e95b49a68f5a1ffb0c56849eef68fc41c0da9f1ff41631703eb20d93befbcaf WatchSource:0}: Error finding container 4e95b49a68f5a1ffb0c56849eef68fc41c0da9f1ff41631703eb20d93befbcaf: Status 404 returned error can't find the container with id 4e95b49a68f5a1ffb0c56849eef68fc41c0da9f1ff41631703eb20d93befbcaf
Jul 23 13:46:48.576553 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:48.576462    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8" event={"ID":"78a34ac0-ed10-48a3-ab9e-deb9e54c8989","Type":"ContainerStarted","Data":"4e95b49a68f5a1ffb0c56849eef68fc41c0da9f1ff41631703eb20d93befbcaf"}
Jul 23 13:46:51.592699 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:51.592662    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8" event={"ID":"78a34ac0-ed10-48a3-ab9e-deb9e54c8989","Type":"ContainerStarted","Data":"8ad93cbf620729d3c3c107b343bf566ddae9ac3acdac628b957769be72bfe682"}
Jul 23 13:46:52.598732 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:52.598693    2568 generic.go:356] "Generic (PLEG): container finished" podID="78a34ac0-ed10-48a3-ab9e-deb9e54c8989" containerID="8ad93cbf620729d3c3c107b343bf566ddae9ac3acdac628b957769be72bfe682" exitCode=0
Jul 23 13:46:52.599233 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:52.598744    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8" event={"ID":"78a34ac0-ed10-48a3-ab9e-deb9e54c8989","Type":"ContainerDied","Data":"8ad93cbf620729d3c3c107b343bf566ddae9ac3acdac628b957769be72bfe682"}
Jul 23 13:46:53.605519 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:53.605484    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8" event={"ID":"78a34ac0-ed10-48a3-ab9e-deb9e54c8989","Type":"ContainerStarted","Data":"bf5968155ec92c0ef158c0f336a1f4c761ee836d388888530c8e32f4a03f6111"}
Jul 23 13:46:53.606004 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:53.605703    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:46:53.634107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:53.634047    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8" podStartSLOduration=2.065229779 podStartE2EDuration="6.634027222s" totalImagesPullingTime="3.498219246s" totalInitContainerRuntime="1.070578197s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:46:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:46:53.631475692 +0000 UTC m=+362.092566730" watchObservedRunningTime="2026-07-23 13:46:53.634027222 +0000 UTC m=+362.095118263"
Jul 23 13:46:58.579132 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:46:58.579101    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6d8d9c69b6-j5swr"
Jul 23 13:47:04.611586 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:04.611553    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c8b88fb69-4snf8"
Jul 23 13:47:10.228698 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.228663    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4"]
Jul 23 13:47:10.241232 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.241211    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4"
Jul 23 13:47:10.243981 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.243958    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 23 13:47:10.244239 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.244216    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 23 13:47:10.244338 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.244281    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4rxz8\""
Jul 23 13:47:10.244338 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.244299    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 23 13:47:10.244816 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.244797    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 23 13:47:10.250193 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.250175    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4"]
Jul 23 13:47:10.355599 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.355560    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca94efb8-d44b-43d6-8d89-ef09eb258bf8-tls-certs\") pod \"kube-auth-proxy-d57cbb598-cfjg4\" (UID: \"ca94efb8-d44b-43d6-8d89-ef09eb258bf8\") " pod="openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4"
Jul 23 13:47:10.355772 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.355639    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqng7\" (UniqueName: \"kubernetes.io/projected/ca94efb8-d44b-43d6-8d89-ef09eb258bf8-kube-api-access-dqng7\") pod \"kube-auth-proxy-d57cbb598-cfjg4\" (UID: \"ca94efb8-d44b-43d6-8d89-ef09eb258bf8\") " pod="openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4"
Jul 23 13:47:10.355772 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.355665    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca94efb8-d44b-43d6-8d89-ef09eb258bf8-tmp\") pod \"kube-auth-proxy-d57cbb598-cfjg4\" (UID: \"ca94efb8-d44b-43d6-8d89-ef09eb258bf8\") " pod="openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4"
Jul 23 13:47:10.456567 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.456526    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca94efb8-d44b-43d6-8d89-ef09eb258bf8-tls-certs\") pod \"kube-auth-proxy-d57cbb598-cfjg4\" (UID: \"ca94efb8-d44b-43d6-8d89-ef09eb258bf8\") " pod="openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4"
Jul 23 13:47:10.456763 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.456619    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqng7\" (UniqueName: \"kubernetes.io/projected/ca94efb8-d44b-43d6-8d89-ef09eb258bf8-kube-api-access-dqng7\") pod \"kube-auth-proxy-d57cbb598-cfjg4\" (UID: \"ca94efb8-d44b-43d6-8d89-ef09eb258bf8\") " pod="openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4"
Jul 23 13:47:10.456763 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.456653    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca94efb8-d44b-43d6-8d89-ef09eb258bf8-tmp\") pod \"kube-auth-proxy-d57cbb598-cfjg4\" (UID: \"ca94efb8-d44b-43d6-8d89-ef09eb258bf8\") " pod="openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4"
Jul 23 13:47:10.458784 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.458760    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca94efb8-d44b-43d6-8d89-ef09eb258bf8-tmp\") pod \"kube-auth-proxy-d57cbb598-cfjg4\" (UID: \"ca94efb8-d44b-43d6-8d89-ef09eb258bf8\") " pod="openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4"
Jul 23 13:47:10.459031 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.459013    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca94efb8-d44b-43d6-8d89-ef09eb258bf8-tls-certs\") pod \"kube-auth-proxy-d57cbb598-cfjg4\" (UID: \"ca94efb8-d44b-43d6-8d89-ef09eb258bf8\") " pod="openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4"
Jul 23 13:47:10.475684 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.475663    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqng7\" (UniqueName: \"kubernetes.io/projected/ca94efb8-d44b-43d6-8d89-ef09eb258bf8-kube-api-access-dqng7\") pod \"kube-auth-proxy-d57cbb598-cfjg4\" (UID: \"ca94efb8-d44b-43d6-8d89-ef09eb258bf8\") " pod="openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4"
Jul 23 13:47:10.553680 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.553651    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4"
Jul 23 13:47:10.691502 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:10.691472    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4"]
Jul 23 13:47:10.694809 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:47:10.694771    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca94efb8_d44b_43d6_8d89_ef09eb258bf8.slice/crio-ce26c0b7c3fddb3259b52ad474723ad71c435bf7239b99546582b12e0f195492 WatchSource:0}: Error finding container ce26c0b7c3fddb3259b52ad474723ad71c435bf7239b99546582b12e0f195492: Status 404 returned error can't find the container with id ce26c0b7c3fddb3259b52ad474723ad71c435bf7239b99546582b12e0f195492
Jul 23 13:47:11.679286 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:11.679252    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4" event={"ID":"ca94efb8-d44b-43d6-8d89-ef09eb258bf8","Type":"ContainerStarted","Data":"ce26c0b7c3fddb3259b52ad474723ad71c435bf7239b99546582b12e0f195492"}
Jul 23 13:47:13.690664 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:13.690576    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4" event={"ID":"ca94efb8-d44b-43d6-8d89-ef09eb258bf8","Type":"ContainerStarted","Data":"203b984ebf54f6f4d6be37e4efd45de847f7f4cc525592ab48adccf4114d7df3"}
Jul 23 13:47:13.709902 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:47:13.709829    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d57cbb598-cfjg4" podStartSLOduration=1.039902933 podStartE2EDuration="3.709812641s" totalImagesPullingTime="2.669909708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:47:13.708491234 +0000 UTC m=+382.169582276" watchObservedRunningTime="2026-07-23 13:47:13.709812641 +0000 UTC m=+382.170903684"
Jul 23 13:48:38.468322 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.468290    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54556fffb6-7p2wc"]
Jul 23 13:48:38.471650 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.471628    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.485292 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.485268    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54556fffb6-7p2wc"]
Jul 23 13:48:38.544756 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.544725    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc078541-1d9c-4eb6-963f-cf4102ade172-console-oauth-config\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.544940 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.544763    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc078541-1d9c-4eb6-963f-cf4102ade172-trusted-ca-bundle\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.544940 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.544783    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc078541-1d9c-4eb6-963f-cf4102ade172-console-config\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.544940 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.544828    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hr5q\" (UniqueName: \"kubernetes.io/projected/fc078541-1d9c-4eb6-963f-cf4102ade172-kube-api-access-9hr5q\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.545109 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.544979    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc078541-1d9c-4eb6-963f-cf4102ade172-console-serving-cert\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.545109 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.545008    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc078541-1d9c-4eb6-963f-cf4102ade172-oauth-serving-cert\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.545109 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.545033    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc078541-1d9c-4eb6-963f-cf4102ade172-service-ca\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.645339 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.645311    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc078541-1d9c-4eb6-963f-cf4102ade172-console-serving-cert\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.645339 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.645338    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc078541-1d9c-4eb6-963f-cf4102ade172-oauth-serving-cert\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.645528 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.645353    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc078541-1d9c-4eb6-963f-cf4102ade172-service-ca\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.645528 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.645380    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc078541-1d9c-4eb6-963f-cf4102ade172-console-oauth-config\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.645528 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.645404    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc078541-1d9c-4eb6-963f-cf4102ade172-trusted-ca-bundle\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.645528 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.645433    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc078541-1d9c-4eb6-963f-cf4102ade172-console-config\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.645702 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.645562    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hr5q\" (UniqueName: \"kubernetes.io/projected/fc078541-1d9c-4eb6-963f-cf4102ade172-kube-api-access-9hr5q\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.646216 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.646192    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc078541-1d9c-4eb6-963f-cf4102ade172-console-config\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.646303 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.646192    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc078541-1d9c-4eb6-963f-cf4102ade172-service-ca\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.646303 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.646221    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc078541-1d9c-4eb6-963f-cf4102ade172-oauth-serving-cert\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.646444 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.646420    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc078541-1d9c-4eb6-963f-cf4102ade172-trusted-ca-bundle\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.647728 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.647710    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc078541-1d9c-4eb6-963f-cf4102ade172-console-serving-cert\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.648354 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.648337    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc078541-1d9c-4eb6-963f-cf4102ade172-console-oauth-config\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.657459 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.657431    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hr5q\" (UniqueName: \"kubernetes.io/projected/fc078541-1d9c-4eb6-963f-cf4102ade172-kube-api-access-9hr5q\") pod \"console-54556fffb6-7p2wc\" (UID: \"fc078541-1d9c-4eb6-963f-cf4102ade172\") " pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.781544 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.781518    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:38.909290 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:38.909268    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54556fffb6-7p2wc"]
Jul 23 13:48:38.912986 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:48:38.912960    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc078541_1d9c_4eb6_963f_cf4102ade172.slice/crio-f1606f7345dc7a3b63ad5616de598941187c9748bf1824423031b377356054bc WatchSource:0}: Error finding container f1606f7345dc7a3b63ad5616de598941187c9748bf1824423031b377356054bc: Status 404 returned error can't find the container with id f1606f7345dc7a3b63ad5616de598941187c9748bf1824423031b377356054bc
Jul 23 13:48:39.043120 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:39.043038    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54556fffb6-7p2wc" event={"ID":"fc078541-1d9c-4eb6-963f-cf4102ade172","Type":"ContainerStarted","Data":"10f63864a973faffa30cc6b752951a5da6442cdf5dc8bbb5aa35f5db992c82db"}
Jul 23 13:48:39.043120 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:39.043089    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54556fffb6-7p2wc" event={"ID":"fc078541-1d9c-4eb6-963f-cf4102ade172","Type":"ContainerStarted","Data":"f1606f7345dc7a3b63ad5616de598941187c9748bf1824423031b377356054bc"}
Jul 23 13:48:39.068428 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:39.068379    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54556fffb6-7p2wc" podStartSLOduration=1.068364418 podStartE2EDuration="1.068364418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:48:39.066894685 +0000 UTC m=+467.527985727" watchObservedRunningTime="2026-07-23 13:48:39.068364418 +0000 UTC m=+467.529455460"
Jul 23 13:48:48.781821 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:48.781784    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:48.782250 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:48.781894    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:48.786458 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:48.786434    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:49.092705 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:49.092624    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54556fffb6-7p2wc"
Jul 23 13:48:49.181940 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:49.181909    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-87d9c6bfd-5d9g2"]
Jul 23 13:48:56.260840 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:56.260807    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m"]
Jul 23 13:48:56.265722 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:56.265700    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m"
Jul 23 13:48:56.270348 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:56.270328    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 23 13:48:56.270472 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:56.270439    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 23 13:48:56.271137 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:56.271115    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 23 13:48:56.271137 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:56.271131    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 23 13:48:56.271318 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:56.271136    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vtgk7\""
Jul 23 13:48:56.275919 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:56.275899    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m"]
Jul 23 13:48:56.397588 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:56.397553    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bbbe4b6-d958-4723-80b9-7c91c4fa5246-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-q6s6m\" (UID: \"9bbbe4b6-d958-4723-80b9-7c91c4fa5246\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m"
Jul 23 13:48:56.397588 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:56.397589    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9bbbe4b6-d958-4723-80b9-7c91c4fa5246-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-q6s6m\" (UID: \"9bbbe4b6-d958-4723-80b9-7c91c4fa5246\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m"
Jul 23 13:48:56.397779 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:56.397623    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpfpz\" (UniqueName: \"kubernetes.io/projected/9bbbe4b6-d958-4723-80b9-7c91c4fa5246-kube-api-access-jpfpz\") pod \"kuadrant-console-plugin-66b5774985-q6s6m\" (UID: \"9bbbe4b6-d958-4723-80b9-7c91c4fa5246\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m"
Jul 23 13:48:56.498380 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:56.498347    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bbbe4b6-d958-4723-80b9-7c91c4fa5246-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-q6s6m\" (UID: \"9bbbe4b6-d958-4723-80b9-7c91c4fa5246\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m"
Jul 23 13:48:56.498544 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:56.498385    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9bbbe4b6-d958-4723-80b9-7c91c4fa5246-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-q6s6m\" (UID: \"9bbbe4b6-d958-4723-80b9-7c91c4fa5246\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m"
Jul 23 13:48:56.498544 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:56.498448    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpfpz\" (UniqueName: \"kubernetes.io/projected/9bbbe4b6-d958-4723-80b9-7c91c4fa5246-kube-api-access-jpfpz\") pod \"kuadrant-console-plugin-66b5774985-q6s6m\" (UID: \"9bbbe4b6-d958-4723-80b9-7c91c4fa5246\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m"
Jul 23 13:48:56.498544 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:48:56.498466    2568 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 23 13:48:56.498703 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:48:56.498548    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bbbe4b6-d958-4723-80b9-7c91c4fa5246-plugin-serving-cert podName:9bbbe4b6-d958-4723-80b9-7c91c4fa5246 nodeName:}" failed. No retries permitted until 2026-07-23 13:48:56.998527186 +0000 UTC m=+485.459618228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/9bbbe4b6-d958-4723-80b9-7c91c4fa5246-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-q6s6m" (UID: "9bbbe4b6-d958-4723-80b9-7c91c4fa5246") : secret "plugin-serving-cert" not found
Jul 23 13:48:56.499059 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:56.499041    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9bbbe4b6-d958-4723-80b9-7c91c4fa5246-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-q6s6m\" (UID: \"9bbbe4b6-d958-4723-80b9-7c91c4fa5246\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m"
Jul 23 13:48:56.518364 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:56.518306    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpfpz\" (UniqueName: \"kubernetes.io/projected/9bbbe4b6-d958-4723-80b9-7c91c4fa5246-kube-api-access-jpfpz\") pod \"kuadrant-console-plugin-66b5774985-q6s6m\" (UID: \"9bbbe4b6-d958-4723-80b9-7c91c4fa5246\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m"
Jul 23 13:48:57.003714 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:57.003687    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bbbe4b6-d958-4723-80b9-7c91c4fa5246-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-q6s6m\" (UID: \"9bbbe4b6-d958-4723-80b9-7c91c4fa5246\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m"
Jul 23 13:48:57.006197 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:57.006173    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bbbe4b6-d958-4723-80b9-7c91c4fa5246-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-q6s6m\" (UID: \"9bbbe4b6-d958-4723-80b9-7c91c4fa5246\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m"
Jul 23 13:48:57.175695 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:57.175662    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m"
Jul 23 13:48:57.318307 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:57.318285    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m"]
Jul 23 13:48:57.320171 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:48:57.320144    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bbbe4b6_d958_4723_80b9_7c91c4fa5246.slice/crio-ed202eb27082e1e672643526a9357f8d59b1277c546ec27f030082f5ec487e8f WatchSource:0}: Error finding container ed202eb27082e1e672643526a9357f8d59b1277c546ec27f030082f5ec487e8f: Status 404 returned error can't find the container with id ed202eb27082e1e672643526a9357f8d59b1277c546ec27f030082f5ec487e8f
Jul 23 13:48:58.131088 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:48:58.131038    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m" event={"ID":"9bbbe4b6-d958-4723-80b9-7c91c4fa5246","Type":"ContainerStarted","Data":"ed202eb27082e1e672643526a9357f8d59b1277c546ec27f030082f5ec487e8f"}
Jul 23 13:49:14.202497 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:14.202447    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-87d9c6bfd-5d9g2" podUID="f0561c6a-fb2d-4479-ae8d-0231a913f826" containerName="console" containerID="cri-o://8c035fcd2b52669314e76dfb40cb686c5da21705e4ef9b67de71d2fafc99c785" gracePeriod=15
Jul 23 13:49:18.196604 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.196579    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:49:18.237616 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.237564    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m" event={"ID":"9bbbe4b6-d958-4723-80b9-7c91c4fa5246","Type":"ContainerStarted","Data":"fa2cd6a9f442f0b2971814e564a2167e6d65b4eff6617064702244608342051b"}
Jul 23 13:49:18.238931 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.238908    2568 generic.go:356] "Generic (PLEG): container finished" podID="f0561c6a-fb2d-4479-ae8d-0231a913f826" containerID="8c035fcd2b52669314e76dfb40cb686c5da21705e4ef9b67de71d2fafc99c785" exitCode=0
Jul 23 13:49:18.239041 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.238942    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87d9c6bfd-5d9g2" event={"ID":"f0561c6a-fb2d-4479-ae8d-0231a913f826","Type":"ContainerDied","Data":"8c035fcd2b52669314e76dfb40cb686c5da21705e4ef9b67de71d2fafc99c785"}
Jul 23 13:49:18.239041 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.238967    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87d9c6bfd-5d9g2"
Jul 23 13:49:18.239041 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.238989    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87d9c6bfd-5d9g2" event={"ID":"f0561c6a-fb2d-4479-ae8d-0231a913f826","Type":"ContainerDied","Data":"4e745192801420efbe020a159b90ef1af4b4c66fc4cae0725e79737476a550f2"}
Jul 23 13:49:18.239041 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.239015    2568 scope.go:117] "RemoveContainer" containerID="8c035fcd2b52669314e76dfb40cb686c5da21705e4ef9b67de71d2fafc99c785"
Jul 23 13:49:18.246813 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.246791    2568 scope.go:117] "RemoveContainer" containerID="8c035fcd2b52669314e76dfb40cb686c5da21705e4ef9b67de71d2fafc99c785"
Jul 23 13:49:18.247088 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:49:18.247069    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c035fcd2b52669314e76dfb40cb686c5da21705e4ef9b67de71d2fafc99c785\": container with ID starting with 8c035fcd2b52669314e76dfb40cb686c5da21705e4ef9b67de71d2fafc99c785 not found: ID does not exist" containerID="8c035fcd2b52669314e76dfb40cb686c5da21705e4ef9b67de71d2fafc99c785"
Jul 23 13:49:18.247146 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.247102    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c035fcd2b52669314e76dfb40cb686c5da21705e4ef9b67de71d2fafc99c785"} err="failed to get container status \"8c035fcd2b52669314e76dfb40cb686c5da21705e4ef9b67de71d2fafc99c785\": rpc error: code = NotFound desc = could not find container \"8c035fcd2b52669314e76dfb40cb686c5da21705e4ef9b67de71d2fafc99c785\": container with ID starting with 8c035fcd2b52669314e76dfb40cb686c5da21705e4ef9b67de71d2fafc99c785 not found: ID does not exist"
Jul 23 13:49:18.255252 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.255213    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q6s6m" podStartSLOduration=1.446828075 podStartE2EDuration="22.255196037s" totalImagesPullingTime="20.808367962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:49:18.254464787 +0000 UTC m=+506.715555829" watchObservedRunningTime="2026-07-23 13:49:18.255196037 +0000 UTC m=+506.716287080"
Jul 23 13:49:18.306026 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.305963    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-config\") pod \"f0561c6a-fb2d-4479-ae8d-0231a913f826\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") "
Jul 23 13:49:18.306026 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.306018    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-oauth-config\") pod \"f0561c6a-fb2d-4479-ae8d-0231a913f826\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") "
Jul 23 13:49:18.306219 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.306048    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td8lq\" (UniqueName: \"kubernetes.io/projected/f0561c6a-fb2d-4479-ae8d-0231a913f826-kube-api-access-td8lq\") pod \"f0561c6a-fb2d-4479-ae8d-0231a913f826\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") "
Jul 23 13:49:18.306219 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.306089    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-service-ca\") pod \"f0561c6a-fb2d-4479-ae8d-0231a913f826\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") "
Jul 23 13:49:18.306219 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.306118    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-trusted-ca-bundle\") pod \"f0561c6a-fb2d-4479-ae8d-0231a913f826\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") "
Jul 23 13:49:18.306219 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.306135    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-serving-cert\") pod \"f0561c6a-fb2d-4479-ae8d-0231a913f826\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") "
Jul 23 13:49:18.306219 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.306150    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-oauth-serving-cert\") pod \"f0561c6a-fb2d-4479-ae8d-0231a913f826\" (UID: \"f0561c6a-fb2d-4479-ae8d-0231a913f826\") "
Jul 23 13:49:18.306462 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.306335    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-config" (OuterVolumeSpecName: "console-config") pod "f0561c6a-fb2d-4479-ae8d-0231a913f826" (UID: "f0561c6a-fb2d-4479-ae8d-0231a913f826"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:49:18.306670 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.306535    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-config\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:49:18.306670 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.306544    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f0561c6a-fb2d-4479-ae8d-0231a913f826" (UID: "f0561c6a-fb2d-4479-ae8d-0231a913f826"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:49:18.306809 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.306737    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-service-ca" (OuterVolumeSpecName: "service-ca") pod "f0561c6a-fb2d-4479-ae8d-0231a913f826" (UID: "f0561c6a-fb2d-4479-ae8d-0231a913f826"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:49:18.306899 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.306874    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f0561c6a-fb2d-4479-ae8d-0231a913f826" (UID: "f0561c6a-fb2d-4479-ae8d-0231a913f826"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 13:49:18.308591 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.308570    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0561c6a-fb2d-4479-ae8d-0231a913f826-kube-api-access-td8lq" (OuterVolumeSpecName: "kube-api-access-td8lq") pod "f0561c6a-fb2d-4479-ae8d-0231a913f826" (UID: "f0561c6a-fb2d-4479-ae8d-0231a913f826"). InnerVolumeSpecName "kube-api-access-td8lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 13:49:18.308710 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.308685    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f0561c6a-fb2d-4479-ae8d-0231a913f826" (UID: "f0561c6a-fb2d-4479-ae8d-0231a913f826"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:49:18.308710 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.308698    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f0561c6a-fb2d-4479-ae8d-0231a913f826" (UID: "f0561c6a-fb2d-4479-ae8d-0231a913f826"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 13:49:18.407514 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.407487    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-oauth-config\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:49:18.407514 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.407510    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td8lq\" (UniqueName: \"kubernetes.io/projected/f0561c6a-fb2d-4479-ae8d-0231a913f826-kube-api-access-td8lq\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:49:18.407514 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.407521    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-service-ca\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:49:18.407704 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.407529    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-trusted-ca-bundle\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:49:18.407704 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.407538    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0561c6a-fb2d-4479-ae8d-0231a913f826-console-serving-cert\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:49:18.407704 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.407548    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0561c6a-fb2d-4479-ae8d-0231a913f826-oauth-serving-cert\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:49:18.568597 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.568539    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-87d9c6bfd-5d9g2"]
Jul 23 13:49:18.573403 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:18.573379    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-87d9c6bfd-5d9g2"]
Jul 23 13:49:19.118703 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:19.118667    2568 patch_prober.go:28] interesting pod/console-87d9c6bfd-5d9g2 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.20:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 23 13:49:19.118906 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:19.118729    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-87d9c6bfd-5d9g2" podUID="f0561c6a-fb2d-4479-ae8d-0231a913f826" containerName="console" probeResult="failure" output="Get \"https://10.134.0.20:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 23 13:49:20.151055 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:20.151025    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0561c6a-fb2d-4479-ae8d-0231a913f826" path="/var/lib/kubelet/pods/f0561c6a-fb2d-4479-ae8d-0231a913f826/volumes"
Jul 23 13:49:42.212511 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.212481    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:49:42.212933 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.212813    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0561c6a-fb2d-4479-ae8d-0231a913f826" containerName="console"
Jul 23 13:49:42.212933 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.212829    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0561c6a-fb2d-4479-ae8d-0231a913f826" containerName="console"
Jul 23 13:49:42.213013 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.212962    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0561c6a-fb2d-4479-ae8d-0231a913f826" containerName="console"
Jul 23 13:49:42.275662 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.275617    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:49:42.275662 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.275663    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:49:42.275889 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.275744    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"
Jul 23 13:49:42.278254 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.278226    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 23 13:49:42.412691 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.412654    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9j4f\" (UniqueName: \"kubernetes.io/projected/6aed8257-e716-4ec6-94ad-07232fffe314-kube-api-access-b9j4f\") pod \"limitador-limitador-54d6d988d5-pbbwv\" (UID: \"6aed8257-e716-4ec6-94ad-07232fffe314\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"
Jul 23 13:49:42.412876 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.412713    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6aed8257-e716-4ec6-94ad-07232fffe314-config-file\") pod \"limitador-limitador-54d6d988d5-pbbwv\" (UID: \"6aed8257-e716-4ec6-94ad-07232fffe314\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"
Jul 23 13:49:42.513526 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.513489    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9j4f\" (UniqueName: \"kubernetes.io/projected/6aed8257-e716-4ec6-94ad-07232fffe314-kube-api-access-b9j4f\") pod \"limitador-limitador-54d6d988d5-pbbwv\" (UID: \"6aed8257-e716-4ec6-94ad-07232fffe314\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"
Jul 23 13:49:42.513672 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.513553    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6aed8257-e716-4ec6-94ad-07232fffe314-config-file\") pod \"limitador-limitador-54d6d988d5-pbbwv\" (UID: \"6aed8257-e716-4ec6-94ad-07232fffe314\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"
Jul 23 13:49:42.514163 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.514146    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6aed8257-e716-4ec6-94ad-07232fffe314-config-file\") pod \"limitador-limitador-54d6d988d5-pbbwv\" (UID: \"6aed8257-e716-4ec6-94ad-07232fffe314\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"
Jul 23 13:49:42.521455 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.521433    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9j4f\" (UniqueName: \"kubernetes.io/projected/6aed8257-e716-4ec6-94ad-07232fffe314-kube-api-access-b9j4f\") pod \"limitador-limitador-54d6d988d5-pbbwv\" (UID: \"6aed8257-e716-4ec6-94ad-07232fffe314\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"
Jul 23 13:49:42.586425 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.586401    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"
Jul 23 13:49:42.723388 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.723293    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:49:42.726201 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:49:42.726176    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aed8257_e716_4ec6_94ad_07232fffe314.slice/crio-2c30403fb4f8bfbdb301386d82be1558b6ada925515b8a757e886a13d52678c6 WatchSource:0}: Error finding container 2c30403fb4f8bfbdb301386d82be1558b6ada925515b8a757e886a13d52678c6: Status 404 returned error can't find the container with id 2c30403fb4f8bfbdb301386d82be1558b6ada925515b8a757e886a13d52678c6
Jul 23 13:49:42.843938 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.843845    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-gh49r"]
Jul 23 13:49:42.848474 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.848458    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gh49r"
Jul 23 13:49:42.850805 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.850786    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-n5d87\""
Jul 23 13:49:42.859138 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.859116    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gh49r"]
Jul 23 13:49:42.917912 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:42.917881    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr9xb\" (UniqueName: \"kubernetes.io/projected/b2d27e9a-af0b-4832-b655-9d43915a287c-kube-api-access-zr9xb\") pod \"authorino-75d556bd77-gh49r\" (UID: \"b2d27e9a-af0b-4832-b655-9d43915a287c\") " pod="kuadrant-system/authorino-75d556bd77-gh49r"
Jul 23 13:49:43.019214 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:43.019177    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr9xb\" (UniqueName: \"kubernetes.io/projected/b2d27e9a-af0b-4832-b655-9d43915a287c-kube-api-access-zr9xb\") pod \"authorino-75d556bd77-gh49r\" (UID: \"b2d27e9a-af0b-4832-b655-9d43915a287c\") " pod="kuadrant-system/authorino-75d556bd77-gh49r"
Jul 23 13:49:43.027574 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:43.027552    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr9xb\" (UniqueName: \"kubernetes.io/projected/b2d27e9a-af0b-4832-b655-9d43915a287c-kube-api-access-zr9xb\") pod \"authorino-75d556bd77-gh49r\" (UID: \"b2d27e9a-af0b-4832-b655-9d43915a287c\") " pod="kuadrant-system/authorino-75d556bd77-gh49r"
Jul 23 13:49:43.158229 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:43.158125    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gh49r"
Jul 23 13:49:43.299873 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:43.299815    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gh49r"]
Jul 23 13:49:43.302906 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:49:43.302881    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2d27e9a_af0b_4832_b655_9d43915a287c.slice/crio-6a760c413b38431b8f5e7d52135dc0c547846efdda9f2b1737653ef698886f23 WatchSource:0}: Error finding container 6a760c413b38431b8f5e7d52135dc0c547846efdda9f2b1737653ef698886f23: Status 404 returned error can't find the container with id 6a760c413b38431b8f5e7d52135dc0c547846efdda9f2b1737653ef698886f23
Jul 23 13:49:43.351571 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:43.351484    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-pbbwv" event={"ID":"6aed8257-e716-4ec6-94ad-07232fffe314","Type":"ContainerStarted","Data":"2c30403fb4f8bfbdb301386d82be1558b6ada925515b8a757e886a13d52678c6"}
Jul 23 13:49:43.353136 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:43.353090    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gh49r" event={"ID":"b2d27e9a-af0b-4832-b655-9d43915a287c","Type":"ContainerStarted","Data":"6a760c413b38431b8f5e7d52135dc0c547846efdda9f2b1737653ef698886f23"}
Jul 23 13:49:47.376240 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:47.376201    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-pbbwv" event={"ID":"6aed8257-e716-4ec6-94ad-07232fffe314","Type":"ContainerStarted","Data":"e02f4a2329bf8f212bba18ef700f54334040e98a4e817846d31abb7174f1fdb2"}
Jul 23 13:49:47.376697 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:47.376301    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"
Jul 23 13:49:47.377694 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:47.377669    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gh49r" event={"ID":"b2d27e9a-af0b-4832-b655-9d43915a287c","Type":"ContainerStarted","Data":"0bcc94e364ce5f8e19ecc24943029786459e0e2363d63dc7e265c93342005b47"}
Jul 23 13:49:47.397157 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:47.397117    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-pbbwv" podStartSLOduration=0.929511804 podStartE2EDuration="5.397104001s" totalImagesPullingTime="4.467592197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:49:47.39595245 +0000 UTC m=+535.857043493" watchObservedRunningTime="2026-07-23 13:49:47.397104001 +0000 UTC m=+535.858195075"
Jul 23 13:49:47.412545 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:47.412494    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-gh49r" podStartSLOduration=1.51913361 podStartE2EDuration="5.412476643s" totalImagesPullingTime="3.893343033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:49:47.411907277 +0000 UTC m=+535.872998320" watchObservedRunningTime="2026-07-23 13:49:47.412476643 +0000 UTC m=+535.873567687"
Jul 23 13:49:58.383776 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:49:58.383745    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"
Jul 23 13:50:18.027156 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:18.027121    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gh49r"]
Jul 23 13:50:18.027621 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:18.027359    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-gh49r" podUID="b2d27e9a-af0b-4832-b655-9d43915a287c" containerName="authorino" containerID="cri-o://0bcc94e364ce5f8e19ecc24943029786459e0e2363d63dc7e265c93342005b47" gracePeriod=30
Jul 23 13:50:18.318398 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:18.318366    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gh49r"
Jul 23 13:50:18.404063 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:18.404030    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr9xb\" (UniqueName: \"kubernetes.io/projected/b2d27e9a-af0b-4832-b655-9d43915a287c-kube-api-access-zr9xb\") pod \"b2d27e9a-af0b-4832-b655-9d43915a287c\" (UID: \"b2d27e9a-af0b-4832-b655-9d43915a287c\") "
Jul 23 13:50:18.406143 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:18.406122    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d27e9a-af0b-4832-b655-9d43915a287c-kube-api-access-zr9xb" (OuterVolumeSpecName: "kube-api-access-zr9xb") pod "b2d27e9a-af0b-4832-b655-9d43915a287c" (UID: "b2d27e9a-af0b-4832-b655-9d43915a287c"). InnerVolumeSpecName "kube-api-access-zr9xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 13:50:18.505100 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:18.505068    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr9xb\" (UniqueName: \"kubernetes.io/projected/b2d27e9a-af0b-4832-b655-9d43915a287c-kube-api-access-zr9xb\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 13:50:18.508476 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:18.508438    2568 generic.go:356] "Generic (PLEG): container finished" podID="b2d27e9a-af0b-4832-b655-9d43915a287c" containerID="0bcc94e364ce5f8e19ecc24943029786459e0e2363d63dc7e265c93342005b47" exitCode=0
Jul 23 13:50:18.508602 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:18.508477    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gh49r"
Jul 23 13:50:18.508602 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:18.508519    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gh49r" event={"ID":"b2d27e9a-af0b-4832-b655-9d43915a287c","Type":"ContainerDied","Data":"0bcc94e364ce5f8e19ecc24943029786459e0e2363d63dc7e265c93342005b47"}
Jul 23 13:50:18.508602 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:18.508559    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gh49r" event={"ID":"b2d27e9a-af0b-4832-b655-9d43915a287c","Type":"ContainerDied","Data":"6a760c413b38431b8f5e7d52135dc0c547846efdda9f2b1737653ef698886f23"}
Jul 23 13:50:18.508602 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:18.508576    2568 scope.go:117] "RemoveContainer" containerID="0bcc94e364ce5f8e19ecc24943029786459e0e2363d63dc7e265c93342005b47"
Jul 23 13:50:18.516709 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:18.516694    2568 scope.go:117] "RemoveContainer" containerID="0bcc94e364ce5f8e19ecc24943029786459e0e2363d63dc7e265c93342005b47"
Jul 23 13:50:18.516970 ip-10-0-138-125 kubenswrapper[2568]: E0723 13:50:18.516953    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bcc94e364ce5f8e19ecc24943029786459e0e2363d63dc7e265c93342005b47\": container with ID starting with 0bcc94e364ce5f8e19ecc24943029786459e0e2363d63dc7e265c93342005b47 not found: ID does not exist" containerID="0bcc94e364ce5f8e19ecc24943029786459e0e2363d63dc7e265c93342005b47"
Jul 23 13:50:18.517035 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:18.516980    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bcc94e364ce5f8e19ecc24943029786459e0e2363d63dc7e265c93342005b47"} err="failed to get container status \"0bcc94e364ce5f8e19ecc24943029786459e0e2363d63dc7e265c93342005b47\": rpc error: code = NotFound desc = could not find container \"0bcc94e364ce5f8e19ecc24943029786459e0e2363d63dc7e265c93342005b47\": container with ID starting with 0bcc94e364ce5f8e19ecc24943029786459e0e2363d63dc7e265c93342005b47 not found: ID does not exist"
Jul 23 13:50:18.546046 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:18.544944    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gh49r"]
Jul 23 13:50:18.550766 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:18.550741    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gh49r"]
Jul 23 13:50:20.151019 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:20.150984    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2d27e9a-af0b-4832-b655-9d43915a287c" path="/var/lib/kubelet/pods/b2d27e9a-af0b-4832-b655-9d43915a287c/volumes"
Jul 23 13:50:59.590371 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:50:59.590337    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:51:27.393380 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:27.393345    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:51:36.496729 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:36.496693    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:51:48.483911 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:48.483876    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:51:57.892274 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:57.892242    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"]
Jul 23 13:51:57.892672 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:57.892578    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2d27e9a-af0b-4832-b655-9d43915a287c" containerName="authorino"
Jul 23 13:51:57.892672 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:57.892591    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2d27e9a-af0b-4832-b655-9d43915a287c" containerName="authorino"
Jul 23 13:51:57.892672 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:57.892647    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2d27e9a-af0b-4832-b655-9d43915a287c" containerName="authorino"
Jul 23 13:51:57.894658 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:57.894641    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:57.897229 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:57.897211    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 23 13:51:57.897229 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:57.897224    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-djm7s\""
Jul 23 13:51:57.897384 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:57.897241    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 23 13:51:57.898057 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:57.898038    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 23 13:51:57.906107 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:57.906087    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"]
Jul 23 13:51:57.922991 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:57.922962    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/581a4447-63d8-4008-8dd0-113e2c1598ca-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:57.923093 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:57.923030    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/581a4447-63d8-4008-8dd0-113e2c1598ca-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:57.923093 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:57.923074    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/581a4447-63d8-4008-8dd0-113e2c1598ca-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:57.923183 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:57.923107    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv4nj\" (UniqueName: \"kubernetes.io/projected/581a4447-63d8-4008-8dd0-113e2c1598ca-kube-api-access-nv4nj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:57.923183 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:57.923141    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/581a4447-63d8-4008-8dd0-113e2c1598ca-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:57.923183 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:57.923156    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/581a4447-63d8-4008-8dd0-113e2c1598ca-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:58.023630 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:58.023596    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/581a4447-63d8-4008-8dd0-113e2c1598ca-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:58.023760 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:58.023645    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/581a4447-63d8-4008-8dd0-113e2c1598ca-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:58.023760 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:58.023673    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/581a4447-63d8-4008-8dd0-113e2c1598ca-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:58.023760 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:58.023699    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv4nj\" (UniqueName: \"kubernetes.io/projected/581a4447-63d8-4008-8dd0-113e2c1598ca-kube-api-access-nv4nj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:58.023760 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:58.023740    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/581a4447-63d8-4008-8dd0-113e2c1598ca-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:58.023986 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:58.023761    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/581a4447-63d8-4008-8dd0-113e2c1598ca-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:58.024083 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:58.024061    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/581a4447-63d8-4008-8dd0-113e2c1598ca-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:58.024159 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:58.024073    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/581a4447-63d8-4008-8dd0-113e2c1598ca-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:58.024214 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:58.024176    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/581a4447-63d8-4008-8dd0-113e2c1598ca-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:58.026023 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:58.025994    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/581a4447-63d8-4008-8dd0-113e2c1598ca-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:58.026205 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:58.026191    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/581a4447-63d8-4008-8dd0-113e2c1598ca-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:58.031576 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:58.031560    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv4nj\" (UniqueName: \"kubernetes.io/projected/581a4447-63d8-4008-8dd0-113e2c1598ca-kube-api-access-nv4nj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf\" (UID: \"581a4447-63d8-4008-8dd0-113e2c1598ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:58.204952 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:58.204885    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:51:58.334979 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:58.334948    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"]
Jul 23 13:51:58.336761 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:51:58.336734    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581a4447_63d8_4008_8dd0_113e2c1598ca.slice/crio-c6a105a00f88921306514489858888dc847681de0bb79c9c4da46a8a9d036a9f WatchSource:0}: Error finding container c6a105a00f88921306514489858888dc847681de0bb79c9c4da46a8a9d036a9f: Status 404 returned error can't find the container with id c6a105a00f88921306514489858888dc847681de0bb79c9c4da46a8a9d036a9f
Jul 23 13:51:58.338144 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:58.338126    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 13:51:58.941489 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:58.941450    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf" event={"ID":"581a4447-63d8-4008-8dd0-113e2c1598ca","Type":"ContainerStarted","Data":"c6a105a00f88921306514489858888dc847681de0bb79c9c4da46a8a9d036a9f"}
Jul 23 13:51:59.391388 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:59.391362    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:51:59.947224 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:59.947188    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf" event={"ID":"581a4447-63d8-4008-8dd0-113e2c1598ca","Type":"ContainerStarted","Data":"bc3b7b3aa9d1cb2d24f2dbce82935cfd5a82ca604ef5aacbb81f7d473c533373"}
Jul 23 13:51:59.966632 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:51:59.966572    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf" podStartSLOduration=1.957261018 podStartE2EDuration="2.966544847s" totalImagesPullingTime="1.009283829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:51:59.966373539 +0000 UTC m=+668.427464581" watchObservedRunningTime="2026-07-23 13:51:59.966544847 +0000 UTC m=+668.427635892"
Jul 23 13:52:08.205227 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:08.205195    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:52:08.205688 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:08.205252    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:52:08.217525 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:08.217502    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:52:08.998063 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:08.998035    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf"
Jul 23 13:52:10.986402 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:10.986366    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:52:20.598373 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.598339    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"]
Jul 23 13:52:20.600715 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.600699    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.603261 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.603237    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 23 13:52:20.613287 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.613258    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"]
Jul 23 13:52:20.708752 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.708710    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020e8ee6-58f1-400f-9649-f9e2aff96419-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.708975 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.708770    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020e8ee6-58f1-400f-9649-f9e2aff96419-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.708975 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.708877    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020e8ee6-58f1-400f-9649-f9e2aff96419-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.708975 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.708924    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020e8ee6-58f1-400f-9649-f9e2aff96419-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.709091 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.708977    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020e8ee6-58f1-400f-9649-f9e2aff96419-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.709091 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.709004    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snjgd\" (UniqueName: \"kubernetes.io/projected/020e8ee6-58f1-400f-9649-f9e2aff96419-kube-api-access-snjgd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.810070 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.810034    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020e8ee6-58f1-400f-9649-f9e2aff96419-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.810070 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.810076    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020e8ee6-58f1-400f-9649-f9e2aff96419-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.810288 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.810171    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020e8ee6-58f1-400f-9649-f9e2aff96419-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.810288 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.810194    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snjgd\" (UniqueName: \"kubernetes.io/projected/020e8ee6-58f1-400f-9649-f9e2aff96419-kube-api-access-snjgd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.810288 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.810226    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020e8ee6-58f1-400f-9649-f9e2aff96419-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.810288 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.810256    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020e8ee6-58f1-400f-9649-f9e2aff96419-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.810571 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.810547    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020e8ee6-58f1-400f-9649-f9e2aff96419-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.810571 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.810560    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020e8ee6-58f1-400f-9649-f9e2aff96419-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.810702 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.810615    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020e8ee6-58f1-400f-9649-f9e2aff96419-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.812442 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.812419    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020e8ee6-58f1-400f-9649-f9e2aff96419-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.812725 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.812705    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020e8ee6-58f1-400f-9649-f9e2aff96419-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.818150 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.818129    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snjgd\" (UniqueName: \"kubernetes.io/projected/020e8ee6-58f1-400f-9649-f9e2aff96419-kube-api-access-snjgd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf\" (UID: \"020e8ee6-58f1-400f-9649-f9e2aff96419\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:20.910255 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:20.910172    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:21.047357 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:21.047330    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"]
Jul 23 13:52:21.048822 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:52:21.048797    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod020e8ee6_58f1_400f_9649_f9e2aff96419.slice/crio-0cd6ec2bd5e3efe9a4610ddcfc3471b15bf76f54d82fa62a159ca652ba4fde3c WatchSource:0}: Error finding container 0cd6ec2bd5e3efe9a4610ddcfc3471b15bf76f54d82fa62a159ca652ba4fde3c: Status 404 returned error can't find the container with id 0cd6ec2bd5e3efe9a4610ddcfc3471b15bf76f54d82fa62a159ca652ba4fde3c
Jul 23 13:52:22.047064 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:22.047019    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf" event={"ID":"020e8ee6-58f1-400f-9649-f9e2aff96419","Type":"ContainerStarted","Data":"7bcd9b3d7974f305b5defe637c1946183b076a3d6d689202a1e904afa61681fd"}
Jul 23 13:52:22.047446 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:22.047079    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf" event={"ID":"020e8ee6-58f1-400f-9649-f9e2aff96419","Type":"ContainerStarted","Data":"0cd6ec2bd5e3efe9a4610ddcfc3471b15bf76f54d82fa62a159ca652ba4fde3c"}
Jul 23 13:52:22.063760 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:22.063710    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf" podStartSLOduration=1.733842134 podStartE2EDuration="2.063696064s" totalImagesPullingTime="329.85393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:52:22.062549768 +0000 UTC m=+690.523640810" watchObservedRunningTime="2026-07-23 13:52:22.063696064 +0000 UTC m=+690.524787084"
Jul 23 13:52:23.377627 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:23.377595    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:52:30.910937 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:30.910892    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:30.910937 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:30.910951    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:30.922808 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:30.922779    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:52:31.096057 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:52:31.096025    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf"
Jul 23 13:53:27.535463 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:27.535424    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f975f7564-rqn6c"]
Jul 23 13:53:27.538843 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:27.538821    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f975f7564-rqn6c"
Jul 23 13:53:27.541988 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:27.541969    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-n5d87\""
Jul 23 13:53:27.542815 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:27.542799    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 23 13:53:27.548332 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:27.548310    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f975f7564-rqn6c"]
Jul 23 13:53:27.688549 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:27.688511    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8a6c465-0bb9-42d9-a063-bc415edb082e-tls-cert\") pod \"authorino-7f975f7564-rqn6c\" (UID: \"e8a6c465-0bb9-42d9-a063-bc415edb082e\") " pod="kuadrant-system/authorino-7f975f7564-rqn6c"
Jul 23 13:53:27.688715 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:27.688607    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg4jv\" (UniqueName: \"kubernetes.io/projected/e8a6c465-0bb9-42d9-a063-bc415edb082e-kube-api-access-vg4jv\") pod \"authorino-7f975f7564-rqn6c\" (UID: \"e8a6c465-0bb9-42d9-a063-bc415edb082e\") " pod="kuadrant-system/authorino-7f975f7564-rqn6c"
Jul 23 13:53:27.789283 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:27.789207    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg4jv\" (UniqueName: \"kubernetes.io/projected/e8a6c465-0bb9-42d9-a063-bc415edb082e-kube-api-access-vg4jv\") pod \"authorino-7f975f7564-rqn6c\" (UID: \"e8a6c465-0bb9-42d9-a063-bc415edb082e\") " pod="kuadrant-system/authorino-7f975f7564-rqn6c"
Jul 23 13:53:27.789283 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:27.789269    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8a6c465-0bb9-42d9-a063-bc415edb082e-tls-cert\") pod \"authorino-7f975f7564-rqn6c\" (UID: \"e8a6c465-0bb9-42d9-a063-bc415edb082e\") " pod="kuadrant-system/authorino-7f975f7564-rqn6c"
Jul 23 13:53:27.791629 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:27.791612    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8a6c465-0bb9-42d9-a063-bc415edb082e-tls-cert\") pod \"authorino-7f975f7564-rqn6c\" (UID: \"e8a6c465-0bb9-42d9-a063-bc415edb082e\") " pod="kuadrant-system/authorino-7f975f7564-rqn6c"
Jul 23 13:53:27.797610 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:27.797583    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg4jv\" (UniqueName: \"kubernetes.io/projected/e8a6c465-0bb9-42d9-a063-bc415edb082e-kube-api-access-vg4jv\") pod \"authorino-7f975f7564-rqn6c\" (UID: \"e8a6c465-0bb9-42d9-a063-bc415edb082e\") " pod="kuadrant-system/authorino-7f975f7564-rqn6c"
Jul 23 13:53:27.848235 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:27.848216    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f975f7564-rqn6c"
Jul 23 13:53:27.981846 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:27.981815    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f975f7564-rqn6c"]
Jul 23 13:53:27.984897 ip-10-0-138-125 kubenswrapper[2568]: W0723 13:53:27.984838    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8a6c465_0bb9_42d9_a063_bc415edb082e.slice/crio-796fc074eab1e7c6575dcb25b5dba6566303833be69d421e0da137a6d50c4a51 WatchSource:0}: Error finding container 796fc074eab1e7c6575dcb25b5dba6566303833be69d421e0da137a6d50c4a51: Status 404 returned error can't find the container with id 796fc074eab1e7c6575dcb25b5dba6566303833be69d421e0da137a6d50c4a51
Jul 23 13:53:28.328032 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:28.328001    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f975f7564-rqn6c" event={"ID":"e8a6c465-0bb9-42d9-a063-bc415edb082e","Type":"ContainerStarted","Data":"796fc074eab1e7c6575dcb25b5dba6566303833be69d421e0da137a6d50c4a51"}
Jul 23 13:53:29.334114 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:29.334072    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f975f7564-rqn6c" event={"ID":"e8a6c465-0bb9-42d9-a063-bc415edb082e","Type":"ContainerStarted","Data":"d93babd26591d6beb12bc71f5aac6426c683840f871fafcfc7e02464dea92978"}
Jul 23 13:53:29.360283 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:29.360236    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7f975f7564-rqn6c" podStartSLOduration=1.908701895 podStartE2EDuration="2.360219416s" totalImagesPullingTime="451.517521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 13:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 13:53:29.357370542 +0000 UTC m=+757.818461585" watchObservedRunningTime="2026-07-23 13:53:29.360219416 +0000 UTC m=+757.821310460"
Jul 23 13:53:56.705618 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:53:56.705580    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:54:07.392384 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:54:07.392342    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:54:18.984409 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:54:18.984319    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:55:58.789040 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:55:58.789002    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:56:10.681751 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:56:10.681708    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:56:45.284519 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:56:45.284486    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:56:55.198004 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:56:55.197965    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:57:04.096121 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:57:04.096080    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:57:15.398140 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:57:15.398064    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:57:25.403252 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:57:25.403219    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:57:36.797331 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:57:36.797298    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:57:47.683616 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:57:47.683580    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:57:57.887457 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:57:57.887422    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:58:23.189934 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:58:23.189902    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:58:34.987672 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:58:34.987635    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:58:58.726871 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:58:58.726785    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:59:09.194113 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:59:09.194077    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:59:26.403363 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:59:26.403325    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:59:35.493795 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:59:35.493760    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:59:45.701751 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:59:45.701711    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 13:59:55.784905 ip-10-0-138-125 kubenswrapper[2568]: I0723 13:59:55.784870    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:00:05.294864 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:00:05.294815    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:01:53.894023 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:01:53.893986    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:02:06.190705 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:02:06.190665    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:02:14.880321 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:02:14.880287    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:02:26.589103 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:02:26.589069    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:02:37.984785 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:02:37.984752    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:02:48.992012 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:02:48.991978    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:02:57.490327 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:02:57.490290    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:03:08.831467 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:03:08.831430    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:03:20.194227 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:03:20.194151    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:03:28.780848 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:03:28.780817    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:03:40.090006 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:03:40.089977    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:04:01.796768 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:01.796732    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:04:11.289595 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:11.289561    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:04:42.700363 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.700328    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"]
Jul 23 14:04:42.703732 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.703712    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.706136 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.706113    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-b10309d1-kserve-self-signed-certs\""
Jul 23 14:04:42.724343 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.724317    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"]
Jul 23 14:04:42.802103 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.802064    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-home\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.802103 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.802106    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-model-cache\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.802318 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.802139    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-tls-certs\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.802318 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.802220    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjpcs\" (UniqueName: \"kubernetes.io/projected/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-kube-api-access-wjpcs\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.802318 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.802241    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-dshm\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.802318 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.802276    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-tmp-dir\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.903675 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.903630    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-dshm\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.903910 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.903696    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-tmp-dir\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.903910 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.903730    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-home\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.903910 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.903760    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-model-cache\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.903910 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.903797    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-tls-certs\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.904087 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.903907    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjpcs\" (UniqueName: \"kubernetes.io/projected/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-kube-api-access-wjpcs\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.904230 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.904207    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-model-cache\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.904289 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.904263    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-home\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.904289 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.904279    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-tmp-dir\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.906052 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.906034    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-dshm\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.906346 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.906327    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-tls-certs\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:42.913053 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:42.913032    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjpcs\" (UniqueName: \"kubernetes.io/projected/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-kube-api-access-wjpcs\") pod \"e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") " pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:43.012067 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:43.012041    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:43.148651 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:43.148626    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"]
Jul 23 14:04:43.151112 ip-10-0-138-125 kubenswrapper[2568]: W0723 14:04:43.151089    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc4ecd8f_131a_4b0d_a6b8_8403f7ddcbb4.slice/crio-c52bf74bddffe32b01e306c79e1d84984dbf1469123a859f2a56f00cb1ffcbb7 WatchSource:0}: Error finding container c52bf74bddffe32b01e306c79e1d84984dbf1469123a859f2a56f00cb1ffcbb7: Status 404 returned error can't find the container with id c52bf74bddffe32b01e306c79e1d84984dbf1469123a859f2a56f00cb1ffcbb7
Jul 23 14:04:43.327672 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:43.327588    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t" event={"ID":"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4","Type":"ContainerStarted","Data":"b7e8bb6089f069a357051f38cc424245269a8925637fa0fe7aafe7122a769168"}
Jul 23 14:04:43.327672 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:43.327637    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t" event={"ID":"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4","Type":"ContainerStarted","Data":"c52bf74bddffe32b01e306c79e1d84984dbf1469123a859f2a56f00cb1ffcbb7"}
Jul 23 14:04:43.347792 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:43.347748    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t" podStartSLOduration=1.347732931 podStartE2EDuration="1.347732931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:04:43.345430494 +0000 UTC m=+1431.806521536" watchObservedRunningTime="2026-07-23 14:04:43.347732931 +0000 UTC m=+1431.808824025"
Jul 23 14:04:45.989206 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:45.989165    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:04:53.012565 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:53.012527    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:53.012960 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:53.012577    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:53.024632 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:53.024606    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:04:53.385948 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:04:53.385868    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:05:01.392679 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.392647    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"]
Jul 23 14:05:01.396088 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.396070    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.399159 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.399141    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-b10309d1-kserve-self-signed-certs\""
Jul 23 14:05:01.411533 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.411502    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"]
Jul 23 14:05:01.465430 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.465403    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd525b1-24c1-4206-b633-83ced30675a2-tls-certs\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.465573 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.465440    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-tmp-dir\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.465573 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.465475    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-model-cache\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.465573 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.465499    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2wmt\" (UniqueName: \"kubernetes.io/projected/0cd525b1-24c1-4206-b633-83ced30675a2-kube-api-access-c2wmt\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.465573 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.465539    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-home\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.465722 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.465608    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-dshm\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.566286 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.566263    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-tmp-dir\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.566407 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.566303    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-model-cache\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.566407 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.566320    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wmt\" (UniqueName: \"kubernetes.io/projected/0cd525b1-24c1-4206-b633-83ced30675a2-kube-api-access-c2wmt\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.566407 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.566374    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-home\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.566518 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.566446    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-dshm\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.566518 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.566480    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd525b1-24c1-4206-b633-83ced30675a2-tls-certs\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.566740 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.566719    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-model-cache\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.566803 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.566729    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-tmp-dir\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.566803 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.566762    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-home\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.568672 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.568644    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-dshm\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.568924 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.568901    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd525b1-24c1-4206-b633-83ced30675a2-tls-certs\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.575412 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.575391    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2wmt\" (UniqueName: \"kubernetes.io/projected/0cd525b1-24c1-4206-b633-83ced30675a2-kube-api-access-c2wmt\") pod \"e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") " pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.704690 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.704614    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:01.842587 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:01.842565    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"]
Jul 23 14:05:01.844458 ip-10-0-138-125 kubenswrapper[2568]: W0723 14:05:01.844426    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cd525b1_24c1_4206_b633_83ced30675a2.slice/crio-69721b6fbc9c9670ddbdb2b0037c881548c32942bf36bb4cceedc911cbf077fe WatchSource:0}: Error finding container 69721b6fbc9c9670ddbdb2b0037c881548c32942bf36bb4cceedc911cbf077fe: Status 404 returned error can't find the container with id 69721b6fbc9c9670ddbdb2b0037c881548c32942bf36bb4cceedc911cbf077fe
Jul 23 14:05:02.417241 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:02.417197    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns" event={"ID":"0cd525b1-24c1-4206-b633-83ced30675a2","Type":"ContainerStarted","Data":"efa5f2c30993aeb70d876333588ca1bcad459067c89de53c5151573781a3369f"}
Jul 23 14:05:02.417642 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:02.417248    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns" event={"ID":"0cd525b1-24c1-4206-b633-83ced30675a2","Type":"ContainerStarted","Data":"69721b6fbc9c9670ddbdb2b0037c881548c32942bf36bb4cceedc911cbf077fe"}
Jul 23 14:05:02.437082 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:02.437035    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns" podStartSLOduration=1.437022042 podStartE2EDuration="1.437022042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:05:02.435717633 +0000 UTC m=+1450.896808676" watchObservedRunningTime="2026-07-23 14:05:02.437022042 +0000 UTC m=+1450.898113084"
Jul 23 14:05:04.916577 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:04.916538    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:05:11.705626 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:11.705584    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:11.705626 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:11.705635    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:11.717799 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:11.717776    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:12.473178 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:12.473146    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:28.367500 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:28.367464    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"]
Jul 23 14:05:30.151126 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:30.151099    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"]
Jul 23 14:05:30.903008 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:30.902977    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:05:43.387367 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.387314    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t" podUID="bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4" containerName="main" containerID="cri-o://b7e8bb6089f069a357051f38cc424245269a8925637fa0fe7aafe7122a769168" gracePeriod=45
Jul 23 14:05:43.607331 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.607297    2568 generic.go:356] "Generic (PLEG): container finished" podID="bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4" containerID="b7e8bb6089f069a357051f38cc424245269a8925637fa0fe7aafe7122a769168" exitCode=0
Jul 23 14:05:43.607514 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.607374    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t" event={"ID":"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4","Type":"ContainerDied","Data":"b7e8bb6089f069a357051f38cc424245269a8925637fa0fe7aafe7122a769168"}
Jul 23 14:05:43.634776 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.634754    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:05:43.732079 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.731998    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-dshm\") pod \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") "
Jul 23 14:05:43.732079 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.732051    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-model-cache\") pod \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") "
Jul 23 14:05:43.732311 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.732084    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-tls-certs\") pod \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") "
Jul 23 14:05:43.732311 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.732138    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-home\") pod \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") "
Jul 23 14:05:43.732311 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.732162    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-tmp-dir\") pod \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") "
Jul 23 14:05:43.732311 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.732197    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjpcs\" (UniqueName: \"kubernetes.io/projected/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-kube-api-access-wjpcs\") pod \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\" (UID: \"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4\") "
Jul 23 14:05:43.732311 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.732295    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4" (UID: "bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:05:43.732564 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.732420    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-home" (OuterVolumeSpecName: "home") pod "bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4" (UID: "bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:05:43.732564 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.732445    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-model-cache\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:05:43.732564 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.732456    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4" (UID: "bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:05:43.734315 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.734286    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-kube-api-access-wjpcs" (OuterVolumeSpecName: "kube-api-access-wjpcs") pod "bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4" (UID: "bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4"). InnerVolumeSpecName "kube-api-access-wjpcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:05:43.734411 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.734331    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4" (UID: "bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:05:43.734411 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.734357    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-dshm" (OuterVolumeSpecName: "dshm") pod "bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4" (UID: "bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:05:43.833631 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.833598    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-tls-certs\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:05:43.833631 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.833625    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-home\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:05:43.833631 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.833636    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-tmp-dir\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:05:43.833887 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.833646    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjpcs\" (UniqueName: \"kubernetes.io/projected/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-kube-api-access-wjpcs\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:05:43.833887 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:43.833654    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4-dshm\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:05:44.612943 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:44.612909    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"
Jul 23 14:05:44.613324 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:44.612946    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t" event={"ID":"bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4","Type":"ContainerDied","Data":"c52bf74bddffe32b01e306c79e1d84984dbf1469123a859f2a56f00cb1ffcbb7"}
Jul 23 14:05:44.613324 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:44.612996    2568 scope.go:117] "RemoveContainer" containerID="b7e8bb6089f069a357051f38cc424245269a8925637fa0fe7aafe7122a769168"
Jul 23 14:05:44.631232 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:44.631202    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"]
Jul 23 14:05:44.636964 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:44.636942    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-b10309d1-kserve-698bb68c75-4gg8t"]
Jul 23 14:05:45.171221 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.171154    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns" podUID="0cd525b1-24c1-4206-b633-83ced30675a2" containerName="main" containerID="cri-o://efa5f2c30993aeb70d876333588ca1bcad459067c89de53c5151573781a3369f" gracePeriod=45
Jul 23 14:05:45.188564 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.188539    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:05:45.412936 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.412909    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:45.447819 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.447764    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-home\") pod \"0cd525b1-24c1-4206-b633-83ced30675a2\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") "
Jul 23 14:05:45.447819 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.447797    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd525b1-24c1-4206-b633-83ced30675a2-tls-certs\") pod \"0cd525b1-24c1-4206-b633-83ced30675a2\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") "
Jul 23 14:05:45.448003 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.447821    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-dshm\") pod \"0cd525b1-24c1-4206-b633-83ced30675a2\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") "
Jul 23 14:05:45.448003 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.447843    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-model-cache\") pod \"0cd525b1-24c1-4206-b633-83ced30675a2\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") "
Jul 23 14:05:45.448003 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.447910    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2wmt\" (UniqueName: \"kubernetes.io/projected/0cd525b1-24c1-4206-b633-83ced30675a2-kube-api-access-c2wmt\") pod \"0cd525b1-24c1-4206-b633-83ced30675a2\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") "
Jul 23 14:05:45.448003 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.447960    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-tmp-dir\") pod \"0cd525b1-24c1-4206-b633-83ced30675a2\" (UID: \"0cd525b1-24c1-4206-b633-83ced30675a2\") "
Jul 23 14:05:45.448206 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.448088    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-home" (OuterVolumeSpecName: "home") pod "0cd525b1-24c1-4206-b633-83ced30675a2" (UID: "0cd525b1-24c1-4206-b633-83ced30675a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:05:45.448206 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.448173    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "0cd525b1-24c1-4206-b633-83ced30675a2" (UID: "0cd525b1-24c1-4206-b633-83ced30675a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:05:45.448306 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.448295    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-home\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:05:45.448353 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.448304    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0cd525b1-24c1-4206-b633-83ced30675a2" (UID: "0cd525b1-24c1-4206-b633-83ced30675a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:05:45.448353 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.448314    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-model-cache\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:05:45.449883 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.449840    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd525b1-24c1-4206-b633-83ced30675a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0cd525b1-24c1-4206-b633-83ced30675a2" (UID: "0cd525b1-24c1-4206-b633-83ced30675a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:05:45.450303 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.450284    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-dshm" (OuterVolumeSpecName: "dshm") pod "0cd525b1-24c1-4206-b633-83ced30675a2" (UID: "0cd525b1-24c1-4206-b633-83ced30675a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:05:45.450303 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.450296    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd525b1-24c1-4206-b633-83ced30675a2-kube-api-access-c2wmt" (OuterVolumeSpecName: "kube-api-access-c2wmt") pod "0cd525b1-24c1-4206-b633-83ced30675a2" (UID: "0cd525b1-24c1-4206-b633-83ced30675a2"). InnerVolumeSpecName "kube-api-access-c2wmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:05:45.549668 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.549640    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd525b1-24c1-4206-b633-83ced30675a2-tls-certs\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:05:45.549668 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.549662    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-dshm\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:05:45.549668 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.549671    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2wmt\" (UniqueName: \"kubernetes.io/projected/0cd525b1-24c1-4206-b633-83ced30675a2-kube-api-access-c2wmt\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:05:45.549876 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.549680    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd525b1-24c1-4206-b633-83ced30675a2-tmp-dir\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:05:45.619372 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.619345    2568 generic.go:356] "Generic (PLEG): container finished" podID="0cd525b1-24c1-4206-b633-83ced30675a2" containerID="efa5f2c30993aeb70d876333588ca1bcad459067c89de53c5151573781a3369f" exitCode=0
Jul 23 14:05:45.619739 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.619386    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"
Jul 23 14:05:45.619739 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.619430    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns" event={"ID":"0cd525b1-24c1-4206-b633-83ced30675a2","Type":"ContainerDied","Data":"efa5f2c30993aeb70d876333588ca1bcad459067c89de53c5151573781a3369f"}
Jul 23 14:05:45.619739 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.619459    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns" event={"ID":"0cd525b1-24c1-4206-b633-83ced30675a2","Type":"ContainerDied","Data":"69721b6fbc9c9670ddbdb2b0037c881548c32942bf36bb4cceedc911cbf077fe"}
Jul 23 14:05:45.619739 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.619475    2568 scope.go:117] "RemoveContainer" containerID="efa5f2c30993aeb70d876333588ca1bcad459067c89de53c5151573781a3369f"
Jul 23 14:05:45.628723 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.628706    2568 scope.go:117] "RemoveContainer" containerID="efa5f2c30993aeb70d876333588ca1bcad459067c89de53c5151573781a3369f"
Jul 23 14:05:45.628980 ip-10-0-138-125 kubenswrapper[2568]: E0723 14:05:45.628962    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa5f2c30993aeb70d876333588ca1bcad459067c89de53c5151573781a3369f\": container with ID starting with efa5f2c30993aeb70d876333588ca1bcad459067c89de53c5151573781a3369f not found: ID does not exist" containerID="efa5f2c30993aeb70d876333588ca1bcad459067c89de53c5151573781a3369f"
Jul 23 14:05:45.629036 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.628987    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa5f2c30993aeb70d876333588ca1bcad459067c89de53c5151573781a3369f"} err="failed to get container status \"efa5f2c30993aeb70d876333588ca1bcad459067c89de53c5151573781a3369f\": rpc error: code = NotFound desc = could not find container \"efa5f2c30993aeb70d876333588ca1bcad459067c89de53c5151573781a3369f\": container with ID starting with efa5f2c30993aeb70d876333588ca1bcad459067c89de53c5151573781a3369f not found: ID does not exist"
Jul 23 14:05:45.640796 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.640776    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"]
Jul 23 14:05:45.644640 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:45.644617    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-b10309d1-kserve-5cbdd9cc9-hzgns"]
Jul 23 14:05:46.151653 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:46.151616    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cd525b1-24c1-4206-b633-83ced30675a2" path="/var/lib/kubelet/pods/0cd525b1-24c1-4206-b633-83ced30675a2/volumes"
Jul 23 14:05:46.152029 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:46.152016    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4" path="/var/lib/kubelet/pods/bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4/volumes"
Jul 23 14:05:55.588876 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:05:55.588819    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:06:05.296353 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:06:05.296316    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:06:19.093970 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:06:19.093888    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:06:28.391750 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:06:28.391715    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:06:38.900226 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:06:38.900191    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:06:47.794676 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:06:47.794642    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:06:57.795260 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:06:57.795226    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:07:10.798066 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:07:10.798029    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:07:20.505596 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:07:20.505554    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:07:30.298561 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:07:30.298526    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:07:40.196714 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:07:40.196677    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:07:52.591048 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:07:52.590973    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:08:00.488338 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:08:00.488300    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:08:12.385603 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:08:12.385566    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:08:24.606277 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:08:24.606241    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:08:37.598488 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:08:37.598455    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:08:46.931133 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:08:46.931088    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:09:08.598960 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:09:08.598923    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:09:24.495133 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:09:24.495101    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:09:56.138411 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:09:56.138380    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:10:04.191079 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:10:04.191042    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:10:39.680880 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:10:39.680832    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:10:50.295505 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:10:50.295468    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:11:23.687326 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:11:23.687290    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:11:32.797707 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:11:32.797669    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:12:08.381979 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:12:08.381944    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:13:35.994194 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:13:35.994153    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:13:47.208884 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:13:47.208782    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:13:59.730536 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:13:59.730501    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:14:19.994294 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:14:19.994262    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:14:59.191748 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:14:59.191713    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:15:17.489725 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.489638    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"]
Jul 23 14:15:17.490193 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.490010    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4" containerName="main"
Jul 23 14:15:17.490193 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.490025    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4" containerName="main"
Jul 23 14:15:17.490193 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.490039    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cd525b1-24c1-4206-b633-83ced30675a2" containerName="main"
Jul 23 14:15:17.490193 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.490045    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cd525b1-24c1-4206-b633-83ced30675a2" containerName="main"
Jul 23 14:15:17.490193 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.490143    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cd525b1-24c1-4206-b633-83ced30675a2" containerName="main"
Jul 23 14:15:17.490193 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.490156    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc4ecd8f-131a-4b0d-a6b8-8403f7ddcbb4" containerName="main"
Jul 23 14:15:17.492926 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.492901    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.495368 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.495341    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b34b8623\"/\"openshift-service-ca.crt\""
Jul 23 14:15:17.496280 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.496260    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b34b8623\"/\"kube-root-ca.crt\""
Jul 23 14:15:17.496280 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.496267    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b34b8623\"/\"default-dockercfg-m97dt\""
Jul 23 14:15:17.496464 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.496260    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b34b8623\"/\"e2e-lifecycle-model-b34b8623-kserve-self-signed-certs\""
Jul 23 14:15:17.506504 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.506472    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"]
Jul 23 14:15:17.576666 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.576639    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhfsr\" (UniqueName: \"kubernetes.io/projected/d066af99-7a37-456d-8b39-8a5a8bddd45c-kube-api-access-vhfsr\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.576801 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.576682    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d066af99-7a37-456d-8b39-8a5a8bddd45c-tls-certs\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.576801 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.576719    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-dshm\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.576801 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.576791    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-home\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.576970 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.576824    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-tmp-dir\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.576970 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.576847    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-model-cache\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.677979 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.677947    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-home\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.678115 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.677990    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-tmp-dir\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.678115 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.678022    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-model-cache\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.678115 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.678085    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhfsr\" (UniqueName: \"kubernetes.io/projected/d066af99-7a37-456d-8b39-8a5a8bddd45c-kube-api-access-vhfsr\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.678278 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.678126    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d066af99-7a37-456d-8b39-8a5a8bddd45c-tls-certs\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.678278 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.678166    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-dshm\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.678397 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.678362    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-tmp-dir\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.678460 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.678393    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-home\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.678460 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.678428    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-model-cache\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.680327 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.680309    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-dshm\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.680504 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.680487    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d066af99-7a37-456d-8b39-8a5a8bddd45c-tls-certs\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.685583 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.685560    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhfsr\" (UniqueName: \"kubernetes.io/projected/d066af99-7a37-456d-8b39-8a5a8bddd45c-kube-api-access-vhfsr\") pod \"e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") " pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.807790 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.807765    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:17.940730 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:17.940707    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"]
Jul 23 14:15:17.942378 ip-10-0-138-125 kubenswrapper[2568]: W0723 14:15:17.942352    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd066af99_7a37_456d_8b39_8a5a8bddd45c.slice/crio-9e43cfc97b0245eadbb99bfd950fedb1628d9bb2a6e52f5cc1c2e0ff90241993 WatchSource:0}: Error finding container 9e43cfc97b0245eadbb99bfd950fedb1628d9bb2a6e52f5cc1c2e0ff90241993: Status 404 returned error can't find the container with id 9e43cfc97b0245eadbb99bfd950fedb1628d9bb2a6e52f5cc1c2e0ff90241993
Jul 23 14:15:18.156977 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:18.156889    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk" event={"ID":"d066af99-7a37-456d-8b39-8a5a8bddd45c","Type":"ContainerStarted","Data":"09a433a5ff396be4db20b2c1b44295a6c8c616e2568d9e1c462552ac9fd62b4d"}
Jul 23 14:15:18.156977 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:18.156941    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk" event={"ID":"d066af99-7a37-456d-8b39-8a5a8bddd45c","Type":"ContainerStarted","Data":"9e43cfc97b0245eadbb99bfd950fedb1628d9bb2a6e52f5cc1c2e0ff90241993"}
Jul 23 14:15:18.176101 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:18.176049    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk" podStartSLOduration=1.176032335 podStartE2EDuration="1.176032335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:15:18.174918115 +0000 UTC m=+2066.636009158" watchObservedRunningTime="2026-07-23 14:15:18.176032335 +0000 UTC m=+2066.637123403"
Jul 23 14:15:21.093617 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:21.093578    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:15:27.808808 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:27.808772    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:27.809273 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:27.808831    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:27.821229 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:27.821204    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:28.209187 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:28.209103    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:15:37.394381 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:37.394334    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:15:55.186617 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:15:55.186581    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:16:05.483779 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:05.483744    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:16:08.253475 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:08.253434    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"]
Jul 23 14:16:23.273489 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.273448    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk" podUID="d066af99-7a37-456d-8b39-8a5a8bddd45c" containerName="main" containerID="cri-o://09a433a5ff396be4db20b2c1b44295a6c8c616e2568d9e1c462552ac9fd62b4d" gracePeriod=45
Jul 23 14:16:23.450320 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.450292    2568 generic.go:356] "Generic (PLEG): container finished" podID="d066af99-7a37-456d-8b39-8a5a8bddd45c" containerID="09a433a5ff396be4db20b2c1b44295a6c8c616e2568d9e1c462552ac9fd62b4d" exitCode=0
Jul 23 14:16:23.450449 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.450361    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk" event={"ID":"d066af99-7a37-456d-8b39-8a5a8bddd45c","Type":"ContainerDied","Data":"09a433a5ff396be4db20b2c1b44295a6c8c616e2568d9e1c462552ac9fd62b4d"}
Jul 23 14:16:23.528786 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.528727    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:16:23.643362 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.643331    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d066af99-7a37-456d-8b39-8a5a8bddd45c-tls-certs\") pod \"d066af99-7a37-456d-8b39-8a5a8bddd45c\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") "
Jul 23 14:16:23.643362 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.643366    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-dshm\") pod \"d066af99-7a37-456d-8b39-8a5a8bddd45c\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") "
Jul 23 14:16:23.643583 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.643423    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-tmp-dir\") pod \"d066af99-7a37-456d-8b39-8a5a8bddd45c\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") "
Jul 23 14:16:23.643633 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.643576    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-home\") pod \"d066af99-7a37-456d-8b39-8a5a8bddd45c\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") "
Jul 23 14:16:23.643672 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.643635    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-model-cache\") pod \"d066af99-7a37-456d-8b39-8a5a8bddd45c\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") "
Jul 23 14:16:23.643718 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.643682    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhfsr\" (UniqueName: \"kubernetes.io/projected/d066af99-7a37-456d-8b39-8a5a8bddd45c-kube-api-access-vhfsr\") pod \"d066af99-7a37-456d-8b39-8a5a8bddd45c\" (UID: \"d066af99-7a37-456d-8b39-8a5a8bddd45c\") "
Jul 23 14:16:23.643761 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.643629    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d066af99-7a37-456d-8b39-8a5a8bddd45c" (UID: "d066af99-7a37-456d-8b39-8a5a8bddd45c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:16:23.643809 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.643790    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-home" (OuterVolumeSpecName: "home") pod "d066af99-7a37-456d-8b39-8a5a8bddd45c" (UID: "d066af99-7a37-456d-8b39-8a5a8bddd45c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:16:23.643938 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.643915    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-model-cache" (OuterVolumeSpecName: "model-cache") pod "d066af99-7a37-456d-8b39-8a5a8bddd45c" (UID: "d066af99-7a37-456d-8b39-8a5a8bddd45c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:16:23.644056 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.644032    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-tmp-dir\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:16:23.644056 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.644057    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-home\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:16:23.644204 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.644071    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-model-cache\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:16:23.645636 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.645618    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-dshm" (OuterVolumeSpecName: "dshm") pod "d066af99-7a37-456d-8b39-8a5a8bddd45c" (UID: "d066af99-7a37-456d-8b39-8a5a8bddd45c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:16:23.645739 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.645716    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d066af99-7a37-456d-8b39-8a5a8bddd45c-kube-api-access-vhfsr" (OuterVolumeSpecName: "kube-api-access-vhfsr") pod "d066af99-7a37-456d-8b39-8a5a8bddd45c" (UID: "d066af99-7a37-456d-8b39-8a5a8bddd45c"). InnerVolumeSpecName "kube-api-access-vhfsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:16:23.645780 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.645735    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d066af99-7a37-456d-8b39-8a5a8bddd45c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d066af99-7a37-456d-8b39-8a5a8bddd45c" (UID: "d066af99-7a37-456d-8b39-8a5a8bddd45c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:16:23.745037 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.745004    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhfsr\" (UniqueName: \"kubernetes.io/projected/d066af99-7a37-456d-8b39-8a5a8bddd45c-kube-api-access-vhfsr\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:16:23.745037 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.745033    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d066af99-7a37-456d-8b39-8a5a8bddd45c-tls-certs\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:16:23.745208 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:23.745047    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d066af99-7a37-456d-8b39-8a5a8bddd45c-dshm\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:16:24.463676 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:24.463595    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk" event={"ID":"d066af99-7a37-456d-8b39-8a5a8bddd45c","Type":"ContainerDied","Data":"9e43cfc97b0245eadbb99bfd950fedb1628d9bb2a6e52f5cc1c2e0ff90241993"}
Jul 23 14:16:24.463676 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:24.463626    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"
Jul 23 14:16:24.463676 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:24.463644    2568 scope.go:117] "RemoveContainer" containerID="09a433a5ff396be4db20b2c1b44295a6c8c616e2568d9e1c462552ac9fd62b4d"
Jul 23 14:16:24.485209 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:24.485188    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"]
Jul 23 14:16:24.489112 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:24.489093    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b34b8623/e2e-lifecycle-model-b34b8623-kserve-7c8b88948c-b82nk"]
Jul 23 14:16:26.151762 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:16:26.151730    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d066af99-7a37-456d-8b39-8a5a8bddd45c" path="/var/lib/kubelet/pods/d066af99-7a37-456d-8b39-8a5a8bddd45c/volumes"
Jul 23 14:17:09.812005 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:17:09.811966    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:17:19.899187 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:17:19.899146    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:18:10.507798 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:10.507761    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:18:19.702213 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:19.702126    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:18:26.883545 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:26.883508    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"]
Jul 23 14:18:26.883953 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:26.883818    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d066af99-7a37-456d-8b39-8a5a8bddd45c" containerName="main"
Jul 23 14:18:26.883953 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:26.883828    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d066af99-7a37-456d-8b39-8a5a8bddd45c" containerName="main"
Jul 23 14:18:26.883953 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:26.883919    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d066af99-7a37-456d-8b39-8a5a8bddd45c" containerName="main"
Jul 23 14:18:26.886491 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:26.886473    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:26.889281 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:26.889258    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-23f4ff\"/\"kube-root-ca.crt\""
Jul 23 14:18:26.890517 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:26.890497    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-23f4ff\"/\"default-dockercfg-74qhk\""
Jul 23 14:18:26.890587 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:26.890503    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-23f4ff\"/\"openshift-service-ca.crt\""
Jul 23 14:18:26.890587 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:26.890529    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-23f4ff\"/\"test-model-23f4ff-kserve-self-signed-certs\""
Jul 23 14:18:26.897259 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:26.897238    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"]
Jul 23 14:18:26.979039 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:26.979014    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9562140-97c7-4022-a55b-fa47fa60d442-tls-certs\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:26.979194 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:26.979045    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-dshm\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:26.979194 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:26.979062    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-model-cache\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:26.979194 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:26.979085    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-tmp-dir\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:26.979298 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:26.979198    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8lsz\" (UniqueName: \"kubernetes.io/projected/f9562140-97c7-4022-a55b-fa47fa60d442-kube-api-access-d8lsz\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:26.979298 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:26.979231    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-home\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:27.079970 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:27.079931    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9562140-97c7-4022-a55b-fa47fa60d442-tls-certs\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:27.079970 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:27.079974    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-dshm\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:27.080211 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:27.079996    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-model-cache\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:27.080211 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:27.080031    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-tmp-dir\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:27.080211 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:27.080091    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8lsz\" (UniqueName: \"kubernetes.io/projected/f9562140-97c7-4022-a55b-fa47fa60d442-kube-api-access-d8lsz\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:27.080211 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:27.080123    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-home\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:27.080460 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:27.080435    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-model-cache\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:27.080528 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:27.080480    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-home\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:27.080528 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:27.080508    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-tmp-dir\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:27.082237 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:27.082210    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-dshm\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:27.082474 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:27.082458    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9562140-97c7-4022-a55b-fa47fa60d442-tls-certs\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:27.089560 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:27.089537    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8lsz\" (UniqueName: \"kubernetes.io/projected/f9562140-97c7-4022-a55b-fa47fa60d442-kube-api-access-d8lsz\") pod \"test-model-23f4ff-kserve-54496f56c5-xvp4x\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") " pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:27.195261 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:27.195181    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:27.326022 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:27.325989    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"]
Jul 23 14:18:27.329074 ip-10-0-138-125 kubenswrapper[2568]: W0723 14:18:27.329048    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9562140_97c7_4022_a55b_fa47fa60d442.slice/crio-0d984f4a0789b6c281af163b64d0cd5094fe71df48abd3b7f0b50682b475e854 WatchSource:0}: Error finding container 0d984f4a0789b6c281af163b64d0cd5094fe71df48abd3b7f0b50682b475e854: Status 404 returned error can't find the container with id 0d984f4a0789b6c281af163b64d0cd5094fe71df48abd3b7f0b50682b475e854
Jul 23 14:18:28.021021 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:28.020983    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x" event={"ID":"f9562140-97c7-4022-a55b-fa47fa60d442","Type":"ContainerStarted","Data":"15e2a727441eec49e1303771e1f1d13e02f050a4801450550eeaeabdf0e91076"}
Jul 23 14:18:28.021399 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:28.021035    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x" event={"ID":"f9562140-97c7-4022-a55b-fa47fa60d442","Type":"ContainerStarted","Data":"0d984f4a0789b6c281af163b64d0cd5094fe71df48abd3b7f0b50682b475e854"}
Jul 23 14:18:28.042290 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:28.042240    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x" podStartSLOduration=2.042220896 podStartE2EDuration="2.042220896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:18:28.041971232 +0000 UTC m=+2256.503062287" watchObservedRunningTime="2026-07-23 14:18:28.042220896 +0000 UTC m=+2256.503311936"
Jul 23 14:18:33.002897 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:33.002860    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:18:37.195796 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:37.195760    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:37.196238 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:37.195815    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:37.209226 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:37.209199    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:18:38.082662 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:18:38.082631    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:19:02.209320 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.209291    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"]
Jul 23 14:19:02.212712 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.212684    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.217232 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.217206    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-80e3ed\"/\"default-dockercfg-8s4h4\""
Jul 23 14:19:02.217678 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.217640    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-80e3ed\"/\"openshift-service-ca.crt\""
Jul 23 14:19:02.218101 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.218077    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-80e3ed\"/\"test-model-80e3ed-kserve-self-signed-certs\""
Jul 23 14:19:02.218460 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.218442    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-80e3ed\"/\"kube-root-ca.crt\""
Jul 23 14:19:02.224606 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.224584    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"]
Jul 23 14:19:02.397360 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.397329    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-tmp-dir\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.397518 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.397381    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-dshm\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.397518 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.397446    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvrpq\" (UniqueName: \"kubernetes.io/projected/2f581693-f5f4-45e3-9019-d1cc04777472-kube-api-access-fvrpq\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.397591 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.397523    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-model-cache\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.397591 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.397562    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-home\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.397922 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.397601    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f581693-f5f4-45e3-9019-d1cc04777472-tls-certs\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.498536 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.498505    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-model-cache\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.498698 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.498546    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-home\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.498698 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.498569    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f581693-f5f4-45e3-9019-d1cc04777472-tls-certs\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.498698 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.498596    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-tmp-dir\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.498698 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.498648    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-dshm\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.498698 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.498691    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvrpq\" (UniqueName: \"kubernetes.io/projected/2f581693-f5f4-45e3-9019-d1cc04777472-kube-api-access-fvrpq\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.499025 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.498999    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-model-cache\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.499090 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.499044    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-home\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.499090 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.499061    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-tmp-dir\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.500967 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.500941    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-dshm\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.501242 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.501223    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f581693-f5f4-45e3-9019-d1cc04777472-tls-certs\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.507806 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.507778    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvrpq\" (UniqueName: \"kubernetes.io/projected/2f581693-f5f4-45e3-9019-d1cc04777472-kube-api-access-fvrpq\") pod \"test-model-80e3ed-kserve-6d76c66475-rm4kg\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") " pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.524626 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.524608    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:02.666105 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:02.666077    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"]
Jul 23 14:19:02.668906 ip-10-0-138-125 kubenswrapper[2568]: W0723 14:19:02.668836    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f581693_f5f4_45e3_9019_d1cc04777472.slice/crio-8918e49c22485eab4a5f29fd4b36e9a5e77164c803bb744b28f1e9e4e69a1a97 WatchSource:0}: Error finding container 8918e49c22485eab4a5f29fd4b36e9a5e77164c803bb744b28f1e9e4e69a1a97: Status 404 returned error can't find the container with id 8918e49c22485eab4a5f29fd4b36e9a5e77164c803bb744b28f1e9e4e69a1a97
Jul 23 14:19:03.179923 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:03.179883    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg" event={"ID":"2f581693-f5f4-45e3-9019-d1cc04777472","Type":"ContainerStarted","Data":"23f58d5e428d7d129cb291af1caf6d7d5e41694608f83ad09de3cb5aef247779"}
Jul 23 14:19:03.179923 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:03.179930    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg" event={"ID":"2f581693-f5f4-45e3-9019-d1cc04777472","Type":"ContainerStarted","Data":"8918e49c22485eab4a5f29fd4b36e9a5e77164c803bb744b28f1e9e4e69a1a97"}
Jul 23 14:19:03.202045 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:03.201994    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg" podStartSLOduration=1.201979185 podStartE2EDuration="1.201979185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:19:03.201659403 +0000 UTC m=+2291.662750446" watchObservedRunningTime="2026-07-23 14:19:03.201979185 +0000 UTC m=+2291.663070226"
Jul 23 14:19:06.294609 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:06.294575    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:19:12.525016 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:12.524980    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:12.525430 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:12.525034    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:12.537540 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:12.537511    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:13.244196 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:13.244165    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:19:19.690195 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:19.690159    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:19:44.271442 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:44.271304    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"]
Jul 23 14:19:45.309815 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:45.309783    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"]
Jul 23 14:19:49.199316 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:49.199277    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:19:59.292101 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.292052    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x" podUID="f9562140-97c7-4022-a55b-fa47fa60d442" containerName="main" containerID="cri-o://15e2a727441eec49e1303771e1f1d13e02f050a4801450550eeaeabdf0e91076" gracePeriod=45
Jul 23 14:19:59.438747 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.438709    2568 generic.go:356] "Generic (PLEG): container finished" podID="f9562140-97c7-4022-a55b-fa47fa60d442" containerID="15e2a727441eec49e1303771e1f1d13e02f050a4801450550eeaeabdf0e91076" exitCode=0
Jul 23 14:19:59.438935 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.438783    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x" event={"ID":"f9562140-97c7-4022-a55b-fa47fa60d442","Type":"ContainerDied","Data":"15e2a727441eec49e1303771e1f1d13e02f050a4801450550eeaeabdf0e91076"}
Jul 23 14:19:59.543900 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.543816    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:19:59.597570 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.597540    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-home\") pod \"f9562140-97c7-4022-a55b-fa47fa60d442\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") "
Jul 23 14:19:59.597728 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.597610    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-dshm\") pod \"f9562140-97c7-4022-a55b-fa47fa60d442\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") "
Jul 23 14:19:59.597728 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.597634    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-model-cache\") pod \"f9562140-97c7-4022-a55b-fa47fa60d442\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") "
Jul 23 14:19:59.597728 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.597670    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8lsz\" (UniqueName: \"kubernetes.io/projected/f9562140-97c7-4022-a55b-fa47fa60d442-kube-api-access-d8lsz\") pod \"f9562140-97c7-4022-a55b-fa47fa60d442\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") "
Jul 23 14:19:59.597728 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.597726    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-tmp-dir\") pod \"f9562140-97c7-4022-a55b-fa47fa60d442\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") "
Jul 23 14:19:59.597997 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.597768    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9562140-97c7-4022-a55b-fa47fa60d442-tls-certs\") pod \"f9562140-97c7-4022-a55b-fa47fa60d442\" (UID: \"f9562140-97c7-4022-a55b-fa47fa60d442\") "
Jul 23 14:19:59.597997 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.597788    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-home" (OuterVolumeSpecName: "home") pod "f9562140-97c7-4022-a55b-fa47fa60d442" (UID: "f9562140-97c7-4022-a55b-fa47fa60d442"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:19:59.597997 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.597929    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9562140-97c7-4022-a55b-fa47fa60d442" (UID: "f9562140-97c7-4022-a55b-fa47fa60d442"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:19:59.598136 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.598045    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9562140-97c7-4022-a55b-fa47fa60d442" (UID: "f9562140-97c7-4022-a55b-fa47fa60d442"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:19:59.598136 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.598063    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-model-cache\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:19:59.598136 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.598083    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-home\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:19:59.599892 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.599864    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-dshm" (OuterVolumeSpecName: "dshm") pod "f9562140-97c7-4022-a55b-fa47fa60d442" (UID: "f9562140-97c7-4022-a55b-fa47fa60d442"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:19:59.600001 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.599931    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9562140-97c7-4022-a55b-fa47fa60d442-kube-api-access-d8lsz" (OuterVolumeSpecName: "kube-api-access-d8lsz") pod "f9562140-97c7-4022-a55b-fa47fa60d442" (UID: "f9562140-97c7-4022-a55b-fa47fa60d442"). InnerVolumeSpecName "kube-api-access-d8lsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:19:59.600048 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.600004    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9562140-97c7-4022-a55b-fa47fa60d442-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9562140-97c7-4022-a55b-fa47fa60d442" (UID: "f9562140-97c7-4022-a55b-fa47fa60d442"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:19:59.699226 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.699195    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-dshm\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:19:59.699226 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.699222    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8lsz\" (UniqueName: \"kubernetes.io/projected/f9562140-97c7-4022-a55b-fa47fa60d442-kube-api-access-d8lsz\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:19:59.699226 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.699233    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9562140-97c7-4022-a55b-fa47fa60d442-tmp-dir\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:19:59.699442 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:19:59.699244    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9562140-97c7-4022-a55b-fa47fa60d442-tls-certs\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:20:00.329522 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.329484    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg" podUID="2f581693-f5f4-45e3-9019-d1cc04777472" containerName="main" containerID="cri-o://23f58d5e428d7d129cb291af1caf6d7d5e41694608f83ad09de3cb5aef247779" gracePeriod=45
Jul 23 14:20:00.444462 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.444433    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"
Jul 23 14:20:00.444598 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.444434    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x" event={"ID":"f9562140-97c7-4022-a55b-fa47fa60d442","Type":"ContainerDied","Data":"0d984f4a0789b6c281af163b64d0cd5094fe71df48abd3b7f0b50682b475e854"}
Jul 23 14:20:00.444598 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.444565    2568 scope.go:117] "RemoveContainer" containerID="15e2a727441eec49e1303771e1f1d13e02f050a4801450550eeaeabdf0e91076"
Jul 23 14:20:00.446328 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.446302    2568 generic.go:356] "Generic (PLEG): container finished" podID="2f581693-f5f4-45e3-9019-d1cc04777472" containerID="23f58d5e428d7d129cb291af1caf6d7d5e41694608f83ad09de3cb5aef247779" exitCode=0
Jul 23 14:20:00.446445 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.446372    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg" event={"ID":"2f581693-f5f4-45e3-9019-d1cc04777472","Type":"ContainerDied","Data":"23f58d5e428d7d129cb291af1caf6d7d5e41694608f83ad09de3cb5aef247779"}
Jul 23 14:20:00.469555 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.469529    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"]
Jul 23 14:20:00.474266 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.474244    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-23f4ff/test-model-23f4ff-kserve-54496f56c5-xvp4x"]
Jul 23 14:20:00.577518 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.577493    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:20:00.607690 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.607665    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-tmp-dir\") pod \"2f581693-f5f4-45e3-9019-d1cc04777472\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") "
Jul 23 14:20:00.607827 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.607711    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-dshm\") pod \"2f581693-f5f4-45e3-9019-d1cc04777472\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") "
Jul 23 14:20:00.607827 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.607732    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-model-cache\") pod \"2f581693-f5f4-45e3-9019-d1cc04777472\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") "
Jul 23 14:20:00.607827 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.607756    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-home\") pod \"2f581693-f5f4-45e3-9019-d1cc04777472\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") "
Jul 23 14:20:00.607827 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.607804    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f581693-f5f4-45e3-9019-d1cc04777472-tls-certs\") pod \"2f581693-f5f4-45e3-9019-d1cc04777472\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") "
Jul 23 14:20:00.608068 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.607844    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvrpq\" (UniqueName: \"kubernetes.io/projected/2f581693-f5f4-45e3-9019-d1cc04777472-kube-api-access-fvrpq\") pod \"2f581693-f5f4-45e3-9019-d1cc04777472\" (UID: \"2f581693-f5f4-45e3-9019-d1cc04777472\") "
Jul 23 14:20:00.608068 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.607977    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f581693-f5f4-45e3-9019-d1cc04777472" (UID: "2f581693-f5f4-45e3-9019-d1cc04777472"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:20:00.608068 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.608028    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f581693-f5f4-45e3-9019-d1cc04777472" (UID: "2f581693-f5f4-45e3-9019-d1cc04777472"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:20:00.608068 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.608051    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-home" (OuterVolumeSpecName: "home") pod "2f581693-f5f4-45e3-9019-d1cc04777472" (UID: "2f581693-f5f4-45e3-9019-d1cc04777472"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:20:00.608285 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.608178    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-tmp-dir\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:20:00.608285 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.608196    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-model-cache\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:20:00.608285 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.608208    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-home\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:20:00.610015 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.609995    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-dshm" (OuterVolumeSpecName: "dshm") pod "2f581693-f5f4-45e3-9019-d1cc04777472" (UID: "2f581693-f5f4-45e3-9019-d1cc04777472"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:20:00.610184 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.610164    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f581693-f5f4-45e3-9019-d1cc04777472-kube-api-access-fvrpq" (OuterVolumeSpecName: "kube-api-access-fvrpq") pod "2f581693-f5f4-45e3-9019-d1cc04777472" (UID: "2f581693-f5f4-45e3-9019-d1cc04777472"). InnerVolumeSpecName "kube-api-access-fvrpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:20:00.610297 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.610276    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f581693-f5f4-45e3-9019-d1cc04777472-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f581693-f5f4-45e3-9019-d1cc04777472" (UID: "2f581693-f5f4-45e3-9019-d1cc04777472"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:20:00.708971 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.708887    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f581693-f5f4-45e3-9019-d1cc04777472-dshm\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:20:00.708971 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.708917    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f581693-f5f4-45e3-9019-d1cc04777472-tls-certs\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:20:00.708971 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:00.708928    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvrpq\" (UniqueName: \"kubernetes.io/projected/2f581693-f5f4-45e3-9019-d1cc04777472-kube-api-access-fvrpq\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:20:01.453459 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:01.453426    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg" event={"ID":"2f581693-f5f4-45e3-9019-d1cc04777472","Type":"ContainerDied","Data":"8918e49c22485eab4a5f29fd4b36e9a5e77164c803bb744b28f1e9e4e69a1a97"}
Jul 23 14:20:01.453459 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:01.453451    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"
Jul 23 14:20:01.454018 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:01.453482    2568 scope.go:117] "RemoveContainer" containerID="23f58d5e428d7d129cb291af1caf6d7d5e41694608f83ad09de3cb5aef247779"
Jul 23 14:20:01.476259 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:01.476235    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"]
Jul 23 14:20:01.483264 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:01.483243    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-80e3ed/test-model-80e3ed-kserve-6d76c66475-rm4kg"]
Jul 23 14:20:02.152796 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:02.152761    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f581693-f5f4-45e3-9019-d1cc04777472" path="/var/lib/kubelet/pods/2f581693-f5f4-45e3-9019-d1cc04777472/volumes"
Jul 23 14:20:02.153281 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:02.153261    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9562140-97c7-4022-a55b-fa47fa60d442" path="/var/lib/kubelet/pods/f9562140-97c7-4022-a55b-fa47fa60d442/volumes"
Jul 23 14:20:04.125116 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:04.125084    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:20:21.390059 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:21.390023    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:20:41.102866 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:41.102825    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:20:53.021396 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:20:53.021356    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:21:17.096527 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:21:17.096439    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:21:42.394130 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:21:42.394091    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:22:00.426271 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:00.426233    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:22:11.112711 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:11.112678    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:22:33.400555 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.400516    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"]
Jul 23 14:22:33.401071 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.401049    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9562140-97c7-4022-a55b-fa47fa60d442" containerName="main"
Jul 23 14:22:33.401120 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.401075    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9562140-97c7-4022-a55b-fa47fa60d442" containerName="main"
Jul 23 14:22:33.401120 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.401109    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f581693-f5f4-45e3-9019-d1cc04777472" containerName="main"
Jul 23 14:22:33.401185 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.401120    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f581693-f5f4-45e3-9019-d1cc04777472" containerName="main"
Jul 23 14:22:33.401218 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.401194    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9562140-97c7-4022-a55b-fa47fa60d442" containerName="main"
Jul 23 14:22:33.401218 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.401205    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f581693-f5f4-45e3-9019-d1cc04777472" containerName="main"
Jul 23 14:22:33.404285 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.404268    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.413346 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.413316    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-42e993\"/\"openshift-service-ca.crt\""
Jul 23 14:22:33.413761 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.413317    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-42e993\"/\"default-dockercfg-tsbkr\""
Jul 23 14:22:33.413761 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.413320    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-42e993\"/\"kube-root-ca.crt\""
Jul 23 14:22:33.413761 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.413423    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-42e993\"/\"auth-test-model-42e993-kserve-self-signed-certs\""
Jul 23 14:22:33.415695 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.415671    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"]
Jul 23 14:22:33.529440 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.529406    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-home\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.529611 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.529456    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-dshm\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.529611 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.529483    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-tmp-dir\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.529611 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.529526    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-model-cache\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.529611 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.529549    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c4dc30-7935-4fca-92b9-b2d2978583e5-tls-certs\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.529611 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.529573    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tqb9\" (UniqueName: \"kubernetes.io/projected/13c4dc30-7935-4fca-92b9-b2d2978583e5-kube-api-access-5tqb9\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.630663 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.630629    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-home\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.630832 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.630684    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-dshm\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.630832 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.630802    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-tmp-dir\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.630950 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.630890    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-model-cache\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.630950 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.630933    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c4dc30-7935-4fca-92b9-b2d2978583e5-tls-certs\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.631052 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.630959    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tqb9\" (UniqueName: \"kubernetes.io/projected/13c4dc30-7935-4fca-92b9-b2d2978583e5-kube-api-access-5tqb9\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.631168 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.631150    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-home\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.631237 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.631223    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-tmp-dir\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.631294 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.631257    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-model-cache\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.632974 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.632955    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-dshm\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.633377 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.633356    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c4dc30-7935-4fca-92b9-b2d2978583e5-tls-certs\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.641724 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.641693    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tqb9\" (UniqueName: \"kubernetes.io/projected/13c4dc30-7935-4fca-92b9-b2d2978583e5-kube-api-access-5tqb9\") pod \"auth-test-model-42e993-kserve-77c9dd4579-j9r62\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") " pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.719632 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.719553    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:33.845878 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:33.845839    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"]
Jul 23 14:22:33.848263 ip-10-0-138-125 kubenswrapper[2568]: W0723 14:22:33.848235    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13c4dc30_7935_4fca_92b9_b2d2978583e5.slice/crio-1dc99a4097833d0763f8dcd6ddbdcdf7e4b12c835e7aa21794e293bb676fe1ec WatchSource:0}: Error finding container 1dc99a4097833d0763f8dcd6ddbdcdf7e4b12c835e7aa21794e293bb676fe1ec: Status 404 returned error can't find the container with id 1dc99a4097833d0763f8dcd6ddbdcdf7e4b12c835e7aa21794e293bb676fe1ec
Jul 23 14:22:34.160280 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:34.160248    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62" event={"ID":"13c4dc30-7935-4fca-92b9-b2d2978583e5","Type":"ContainerStarted","Data":"4160db157f1f3f967e4f6738d858ec5b1285a493776e4b23d5b3f46e5c3f482c"}
Jul 23 14:22:34.160280 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:34.160288    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62" event={"ID":"13c4dc30-7935-4fca-92b9-b2d2978583e5","Type":"ContainerStarted","Data":"1dc99a4097833d0763f8dcd6ddbdcdf7e4b12c835e7aa21794e293bb676fe1ec"}
Jul 23 14:22:34.178063 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:34.178018    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62" podStartSLOduration=1.178003533 podStartE2EDuration="1.178003533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:22:34.176264528 +0000 UTC m=+2502.637355584" watchObservedRunningTime="2026-07-23 14:22:34.178003533 +0000 UTC m=+2502.639094588"
Jul 23 14:22:36.494260 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:36.494221    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:22:43.720360 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:43.720269    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:43.720360 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:43.720328    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:43.732380 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:43.732355    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:44.212983 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:44.212946    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:22:52.237495 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.237462    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"]
Jul 23 14:22:52.241061 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.241040    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.244085 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.244063    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-52507a\"/\"kube-root-ca.crt\""
Jul 23 14:22:52.244213 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.244127    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-52507a\"/\"openshift-service-ca.crt\""
Jul 23 14:22:52.245128 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.245106    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-52507a\"/\"default-dockercfg-rn8zk\""
Jul 23 14:22:52.245297 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.245278    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-52507a\"/\"auth-test-model-52507a-kserve-self-signed-certs\""
Jul 23 14:22:52.257837 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.257817    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"]
Jul 23 14:22:52.398364 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.398327    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-model-cache\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.398364 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.398363    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eca5fe0-c440-480d-8d41-78f93e76e8cd-tls-certs\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.398610 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.398404    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-dshm\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.398610 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.398438    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-tmp-dir\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.398610 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.398460    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-home\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.398610 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.398526    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf6hq\" (UniqueName: \"kubernetes.io/projected/7eca5fe0-c440-480d-8d41-78f93e76e8cd-kube-api-access-cf6hq\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.499520 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.499493    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-model-cache\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.499693 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.499530    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eca5fe0-c440-480d-8d41-78f93e76e8cd-tls-certs\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.499693 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.499575    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-dshm\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.499693 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.499611    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-tmp-dir\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.499693 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.499645    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-home\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.499693 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.499683    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6hq\" (UniqueName: \"kubernetes.io/projected/7eca5fe0-c440-480d-8d41-78f93e76e8cd-kube-api-access-cf6hq\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.500012 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.499990    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-model-cache\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.500071 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.500022    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-tmp-dir\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.500125 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.500101    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-home\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.501884 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.501839    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-dshm\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.502035 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.502018    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eca5fe0-c440-480d-8d41-78f93e76e8cd-tls-certs\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.509738 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.509713    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6hq\" (UniqueName: \"kubernetes.io/projected/7eca5fe0-c440-480d-8d41-78f93e76e8cd-kube-api-access-cf6hq\") pod \"auth-test-model-52507a-kserve-7c84985899-tc7mj\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") " pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.549258 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.549227    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:22:52.686097 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:52.685951    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"]
Jul 23 14:22:52.688418 ip-10-0-138-125 kubenswrapper[2568]: W0723 14:22:52.688387    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7eca5fe0_c440_480d_8d41_78f93e76e8cd.slice/crio-9aeb46f1d1fda715ecdc61d7aa0341256e5b19c52d8e36d7a71e1cb35c17c217 WatchSource:0}: Error finding container 9aeb46f1d1fda715ecdc61d7aa0341256e5b19c52d8e36d7a71e1cb35c17c217: Status 404 returned error can't find the container with id 9aeb46f1d1fda715ecdc61d7aa0341256e5b19c52d8e36d7a71e1cb35c17c217
Jul 23 14:22:53.243226 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:53.243187    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj" event={"ID":"7eca5fe0-c440-480d-8d41-78f93e76e8cd","Type":"ContainerStarted","Data":"630e4f6fbf17b1d5891ef4d838e5230a0c17c6c3710b0ed690e919a05c89e7c7"}
Jul 23 14:22:53.243226 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:53.243229    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj" event={"ID":"7eca5fe0-c440-480d-8d41-78f93e76e8cd","Type":"ContainerStarted","Data":"9aeb46f1d1fda715ecdc61d7aa0341256e5b19c52d8e36d7a71e1cb35c17c217"}
Jul 23 14:22:53.268876 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:53.268809    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj" podStartSLOduration=1.268794885 podStartE2EDuration="1.268794885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:22:53.266268365 +0000 UTC m=+2521.727359401" watchObservedRunningTime="2026-07-23 14:22:53.268794885 +0000 UTC m=+2521.729885927"
Jul 23 14:22:55.706509 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:22:55.706472    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:23:02.550035 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:23:02.549998    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:23:02.550447 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:23:02.550094    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:23:02.562255 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:23:02.562228    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:23:03.300884 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:23:03.300841    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:23:22.152747 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:23:22.152712    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:23:40.999815 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:23:40.999780    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:23:56.235359 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:23:56.235324    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:24:15.909005 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:24:15.908915    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:24:34.900518 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:24:34.900483    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:24:53.996597 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:24:53.996560    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:25:09.966153 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:09.966110    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"]
Jul 23 14:25:10.825940 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:10.825904    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"]
Jul 23 14:25:13.294841 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:13.294807    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:25:24.986101 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:24.986060    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62" podUID="13c4dc30-7935-4fca-92b9-b2d2978583e5" containerName="main" containerID="cri-o://4160db157f1f3f967e4f6738d858ec5b1285a493776e4b23d5b3f46e5c3f482c" gracePeriod=45
Jul 23 14:25:25.232668 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.232643    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:25:25.408682 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.408647    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-model-cache\") pod \"13c4dc30-7935-4fca-92b9-b2d2978583e5\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") "
Jul 23 14:25:25.408901 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.408709    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c4dc30-7935-4fca-92b9-b2d2978583e5-tls-certs\") pod \"13c4dc30-7935-4fca-92b9-b2d2978583e5\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") "
Jul 23 14:25:25.408901 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.408759    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-tmp-dir\") pod \"13c4dc30-7935-4fca-92b9-b2d2978583e5\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") "
Jul 23 14:25:25.408901 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.408798    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-home\") pod \"13c4dc30-7935-4fca-92b9-b2d2978583e5\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") "
Jul 23 14:25:25.408901 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.408837    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tqb9\" (UniqueName: \"kubernetes.io/projected/13c4dc30-7935-4fca-92b9-b2d2978583e5-kube-api-access-5tqb9\") pod \"13c4dc30-7935-4fca-92b9-b2d2978583e5\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") "
Jul 23 14:25:25.408901 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.408890    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-dshm\") pod \"13c4dc30-7935-4fca-92b9-b2d2978583e5\" (UID: \"13c4dc30-7935-4fca-92b9-b2d2978583e5\") "
Jul 23 14:25:25.409193 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.408967    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "13c4dc30-7935-4fca-92b9-b2d2978583e5" (UID: "13c4dc30-7935-4fca-92b9-b2d2978583e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:25:25.409193 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.409070    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-home" (OuterVolumeSpecName: "home") pod "13c4dc30-7935-4fca-92b9-b2d2978583e5" (UID: "13c4dc30-7935-4fca-92b9-b2d2978583e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:25:25.409193 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.409077    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13c4dc30-7935-4fca-92b9-b2d2978583e5" (UID: "13c4dc30-7935-4fca-92b9-b2d2978583e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:25:25.409193 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.409107    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-model-cache\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:25:25.411132 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.411096    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c4dc30-7935-4fca-92b9-b2d2978583e5-kube-api-access-5tqb9" (OuterVolumeSpecName: "kube-api-access-5tqb9") pod "13c4dc30-7935-4fca-92b9-b2d2978583e5" (UID: "13c4dc30-7935-4fca-92b9-b2d2978583e5"). InnerVolumeSpecName "kube-api-access-5tqb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:25:25.411238 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.411127    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-dshm" (OuterVolumeSpecName: "dshm") pod "13c4dc30-7935-4fca-92b9-b2d2978583e5" (UID: "13c4dc30-7935-4fca-92b9-b2d2978583e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:25:25.411238 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.411166    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c4dc30-7935-4fca-92b9-b2d2978583e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13c4dc30-7935-4fca-92b9-b2d2978583e5" (UID: "13c4dc30-7935-4fca-92b9-b2d2978583e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:25:25.510341 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.510311    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-home\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:25:25.510341 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.510334    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tqb9\" (UniqueName: \"kubernetes.io/projected/13c4dc30-7935-4fca-92b9-b2d2978583e5-kube-api-access-5tqb9\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:25:25.510341 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.510344    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-dshm\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:25:25.510550 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.510352    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c4dc30-7935-4fca-92b9-b2d2978583e5-tls-certs\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:25:25.510550 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.510380    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c4dc30-7935-4fca-92b9-b2d2978583e5-tmp-dir\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:25:25.792979 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.792940    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:25:25.845757 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.845706    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj" podUID="7eca5fe0-c440-480d-8d41-78f93e76e8cd" containerName="main" containerID="cri-o://630e4f6fbf17b1d5891ef4d838e5230a0c17c6c3710b0ed690e919a05c89e7c7" gracePeriod=45
Jul 23 14:25:25.947412 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.947382    2568 generic.go:356] "Generic (PLEG): container finished" podID="13c4dc30-7935-4fca-92b9-b2d2978583e5" containerID="4160db157f1f3f967e4f6738d858ec5b1285a493776e4b23d5b3f46e5c3f482c" exitCode=0
Jul 23 14:25:25.947532 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.947427    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"
Jul 23 14:25:25.947532 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.947464    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62" event={"ID":"13c4dc30-7935-4fca-92b9-b2d2978583e5","Type":"ContainerDied","Data":"4160db157f1f3f967e4f6738d858ec5b1285a493776e4b23d5b3f46e5c3f482c"}
Jul 23 14:25:25.947532 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.947501    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62" event={"ID":"13c4dc30-7935-4fca-92b9-b2d2978583e5","Type":"ContainerDied","Data":"1dc99a4097833d0763f8dcd6ddbdcdf7e4b12c835e7aa21794e293bb676fe1ec"}
Jul 23 14:25:25.947532 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.947516    2568 scope.go:117] "RemoveContainer" containerID="4160db157f1f3f967e4f6738d858ec5b1285a493776e4b23d5b3f46e5c3f482c"
Jul 23 14:25:25.956117 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.956100    2568 scope.go:117] "RemoveContainer" containerID="4160db157f1f3f967e4f6738d858ec5b1285a493776e4b23d5b3f46e5c3f482c"
Jul 23 14:25:25.956368 ip-10-0-138-125 kubenswrapper[2568]: E0723 14:25:25.956352    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4160db157f1f3f967e4f6738d858ec5b1285a493776e4b23d5b3f46e5c3f482c\": container with ID starting with 4160db157f1f3f967e4f6738d858ec5b1285a493776e4b23d5b3f46e5c3f482c not found: ID does not exist" containerID="4160db157f1f3f967e4f6738d858ec5b1285a493776e4b23d5b3f46e5c3f482c"
Jul 23 14:25:25.956425 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.956377    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4160db157f1f3f967e4f6738d858ec5b1285a493776e4b23d5b3f46e5c3f482c"} err="failed to get container status \"4160db157f1f3f967e4f6738d858ec5b1285a493776e4b23d5b3f46e5c3f482c\": rpc error: code = NotFound desc = could not find container \"4160db157f1f3f967e4f6738d858ec5b1285a493776e4b23d5b3f46e5c3f482c\": container with ID starting with 4160db157f1f3f967e4f6738d858ec5b1285a493776e4b23d5b3f46e5c3f482c not found: ID does not exist"
Jul 23 14:25:25.968612 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.968589    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"]
Jul 23 14:25:25.972413 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:25.972393    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-42e993/auth-test-model-42e993-kserve-77c9dd4579-j9r62"]
Jul 23 14:25:26.095329 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.095298    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:25:26.150990 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.150965    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13c4dc30-7935-4fca-92b9-b2d2978583e5" path="/var/lib/kubelet/pods/13c4dc30-7935-4fca-92b9-b2d2978583e5/volumes"
Jul 23 14:25:26.216358 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.216325    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-model-cache\") pod \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") "
Jul 23 14:25:26.216532 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.216373    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-dshm\") pod \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") "
Jul 23 14:25:26.216532 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.216396    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eca5fe0-c440-480d-8d41-78f93e76e8cd-tls-certs\") pod \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") "
Jul 23 14:25:26.216532 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.216418    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-home\") pod \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") "
Jul 23 14:25:26.216532 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.216482    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-tmp-dir\") pod \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") "
Jul 23 14:25:26.216532 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.216515    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf6hq\" (UniqueName: \"kubernetes.io/projected/7eca5fe0-c440-480d-8d41-78f93e76e8cd-kube-api-access-cf6hq\") pod \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\" (UID: \"7eca5fe0-c440-480d-8d41-78f93e76e8cd\") "
Jul 23 14:25:26.216805 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.216613    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "7eca5fe0-c440-480d-8d41-78f93e76e8cd" (UID: "7eca5fe0-c440-480d-8d41-78f93e76e8cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:25:26.216805 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.216740    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-model-cache\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:25:26.216805 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.216752    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-home" (OuterVolumeSpecName: "home") pod "7eca5fe0-c440-480d-8d41-78f93e76e8cd" (UID: "7eca5fe0-c440-480d-8d41-78f93e76e8cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:25:26.216805 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.216772    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7eca5fe0-c440-480d-8d41-78f93e76e8cd" (UID: "7eca5fe0-c440-480d-8d41-78f93e76e8cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:25:26.218790 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.218761    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-dshm" (OuterVolumeSpecName: "dshm") pod "7eca5fe0-c440-480d-8d41-78f93e76e8cd" (UID: "7eca5fe0-c440-480d-8d41-78f93e76e8cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:25:26.218907 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.218798    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eca5fe0-c440-480d-8d41-78f93e76e8cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7eca5fe0-c440-480d-8d41-78f93e76e8cd" (UID: "7eca5fe0-c440-480d-8d41-78f93e76e8cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:25:26.218907 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.218833    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eca5fe0-c440-480d-8d41-78f93e76e8cd-kube-api-access-cf6hq" (OuterVolumeSpecName: "kube-api-access-cf6hq") pod "7eca5fe0-c440-480d-8d41-78f93e76e8cd" (UID: "7eca5fe0-c440-480d-8d41-78f93e76e8cd"). InnerVolumeSpecName "kube-api-access-cf6hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:25:26.318196 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.318168    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf6hq\" (UniqueName: \"kubernetes.io/projected/7eca5fe0-c440-480d-8d41-78f93e76e8cd-kube-api-access-cf6hq\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:25:26.318196 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.318194    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-dshm\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:25:26.318372 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.318204    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eca5fe0-c440-480d-8d41-78f93e76e8cd-tls-certs\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:25:26.318372 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.318215    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-home\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:25:26.318372 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.318223    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eca5fe0-c440-480d-8d41-78f93e76e8cd-tmp-dir\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:25:26.954425 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.954389    2568 generic.go:356] "Generic (PLEG): container finished" podID="7eca5fe0-c440-480d-8d41-78f93e76e8cd" containerID="630e4f6fbf17b1d5891ef4d838e5230a0c17c6c3710b0ed690e919a05c89e7c7" exitCode=0
Jul 23 14:25:26.954623 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.954436    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"
Jul 23 14:25:26.954623 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.954485    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj" event={"ID":"7eca5fe0-c440-480d-8d41-78f93e76e8cd","Type":"ContainerDied","Data":"630e4f6fbf17b1d5891ef4d838e5230a0c17c6c3710b0ed690e919a05c89e7c7"}
Jul 23 14:25:26.954623 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.954535    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj" event={"ID":"7eca5fe0-c440-480d-8d41-78f93e76e8cd","Type":"ContainerDied","Data":"9aeb46f1d1fda715ecdc61d7aa0341256e5b19c52d8e36d7a71e1cb35c17c217"}
Jul 23 14:25:26.954623 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.954559    2568 scope.go:117] "RemoveContainer" containerID="630e4f6fbf17b1d5891ef4d838e5230a0c17c6c3710b0ed690e919a05c89e7c7"
Jul 23 14:25:26.962993 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.962978    2568 scope.go:117] "RemoveContainer" containerID="630e4f6fbf17b1d5891ef4d838e5230a0c17c6c3710b0ed690e919a05c89e7c7"
Jul 23 14:25:26.963218 ip-10-0-138-125 kubenswrapper[2568]: E0723 14:25:26.963202    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"630e4f6fbf17b1d5891ef4d838e5230a0c17c6c3710b0ed690e919a05c89e7c7\": container with ID starting with 630e4f6fbf17b1d5891ef4d838e5230a0c17c6c3710b0ed690e919a05c89e7c7 not found: ID does not exist" containerID="630e4f6fbf17b1d5891ef4d838e5230a0c17c6c3710b0ed690e919a05c89e7c7"
Jul 23 14:25:26.963271 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.963224    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630e4f6fbf17b1d5891ef4d838e5230a0c17c6c3710b0ed690e919a05c89e7c7"} err="failed to get container status \"630e4f6fbf17b1d5891ef4d838e5230a0c17c6c3710b0ed690e919a05c89e7c7\": rpc error: code = NotFound desc = could not find container \"630e4f6fbf17b1d5891ef4d838e5230a0c17c6c3710b0ed690e919a05c89e7c7\": container with ID starting with 630e4f6fbf17b1d5891ef4d838e5230a0c17c6c3710b0ed690e919a05c89e7c7 not found: ID does not exist"
Jul 23 14:25:26.974126 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.974101    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"]
Jul 23 14:25:26.978378 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:26.978358    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-52507a/auth-test-model-52507a-kserve-7c84985899-tc7mj"]
Jul 23 14:25:28.151135 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.151107    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eca5fe0-c440-480d-8d41-78f93e76e8cd" path="/var/lib/kubelet/pods/7eca5fe0-c440-480d-8d41-78f93e76e8cd/volumes"
Jul 23 14:25:28.791495 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.791465    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"]
Jul 23 14:25:28.791841 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.791827    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c4dc30-7935-4fca-92b9-b2d2978583e5" containerName="main"
Jul 23 14:25:28.791907 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.791844    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c4dc30-7935-4fca-92b9-b2d2978583e5" containerName="main"
Jul 23 14:25:28.791907 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.791870    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eca5fe0-c440-480d-8d41-78f93e76e8cd" containerName="main"
Jul 23 14:25:28.791907 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.791877    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eca5fe0-c440-480d-8d41-78f93e76e8cd" containerName="main"
Jul 23 14:25:28.791999 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.791938    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eca5fe0-c440-480d-8d41-78f93e76e8cd" containerName="main"
Jul 23 14:25:28.791999 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.791954    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="13c4dc30-7935-4fca-92b9-b2d2978583e5" containerName="main"
Jul 23 14:25:28.794835 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.794816    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:28.798256 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.798234    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-42e993\"/\"kube-root-ca.crt\""
Jul 23 14:25:28.798256 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.798251    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-42e993\"/\"openshift-service-ca.crt\""
Jul 23 14:25:28.798381 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.798234    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-42e993\"/\"default-dockercfg-tsbkr\""
Jul 23 14:25:28.798381 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.798235    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-42e993\"/\"sub-test-model-42e993-kserve-self-signed-certs\""
Jul 23 14:25:28.810357 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.810332    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"]
Jul 23 14:25:28.940062 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.940032    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-dshm\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:28.940193 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.940071    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bzhg\" (UniqueName: \"kubernetes.io/projected/6e78a980-2627-44e6-8e5e-efe403844bdd-kube-api-access-6bzhg\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:28.940193 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.940110    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-home\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:28.940193 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.940128    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-tmp-dir\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:28.940193 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.940143    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e78a980-2627-44e6-8e5e-efe403844bdd-tls-certs\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:28.940375 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:28.940196    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-model-cache\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:29.041163 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.041130    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-home\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:29.041163 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.041160    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-tmp-dir\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:29.041352 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.041175    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e78a980-2627-44e6-8e5e-efe403844bdd-tls-certs\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:29.041352 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.041205    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-model-cache\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:29.041352 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.041235    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-dshm\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:29.041352 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.041259    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bzhg\" (UniqueName: \"kubernetes.io/projected/6e78a980-2627-44e6-8e5e-efe403844bdd-kube-api-access-6bzhg\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:29.041635 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.041578    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-home\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:29.041635 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.041619    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-tmp-dir\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:29.041728 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.041642    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-model-cache\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:29.043579 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.043554    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-dshm\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:29.043796 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.043776    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e78a980-2627-44e6-8e5e-efe403844bdd-tls-certs\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:29.049261 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.049241    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bzhg\" (UniqueName: \"kubernetes.io/projected/6e78a980-2627-44e6-8e5e-efe403844bdd-kube-api-access-6bzhg\") pod \"sub-test-model-42e993-kserve-866788d9dc-bvss7\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") " pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:29.109880 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.109839    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:29.245066 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.245043    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"]
Jul 23 14:25:29.247003 ip-10-0-138-125 kubenswrapper[2568]: W0723 14:25:29.246976    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e78a980_2627_44e6_8e5e_efe403844bdd.slice/crio-35fd6c86ed840a10430950c709869f12350df9d5c5d604d519081cf7a9d736fe WatchSource:0}: Error finding container 35fd6c86ed840a10430950c709869f12350df9d5c5d604d519081cf7a9d736fe: Status 404 returned error can't find the container with id 35fd6c86ed840a10430950c709869f12350df9d5c5d604d519081cf7a9d736fe
Jul 23 14:25:29.969681 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.969648    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7" event={"ID":"6e78a980-2627-44e6-8e5e-efe403844bdd","Type":"ContainerStarted","Data":"82f5e2ac883f810ee6b51129a5085228469d61f0e36e4fd97db19562d450a3cf"}
Jul 23 14:25:29.969681 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.969684    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7" event={"ID":"6e78a980-2627-44e6-8e5e-efe403844bdd","Type":"ContainerStarted","Data":"35fd6c86ed840a10430950c709869f12350df9d5c5d604d519081cf7a9d736fe"}
Jul 23 14:25:29.989375 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:29.989331    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7" podStartSLOduration=1.9893137379999999 podStartE2EDuration="1.989313738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:25:29.988429874 +0000 UTC m=+2678.449520921" watchObservedRunningTime="2026-07-23 14:25:29.989313738 +0000 UTC m=+2678.450404780"
Jul 23 14:25:33.285744 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:33.285710    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:25:39.110766 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:39.110732    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:39.111242 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:39.110783    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:39.123143 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:39.123120    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:40.024817 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:40.024783    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:25:46.776577 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:46.776497    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"]
Jul 23 14:25:46.780098 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:46.780081    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:46.783835 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:46.783800    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-52507a\"/\"openshift-service-ca.crt\""
Jul 23 14:25:46.783835 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:46.783826    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-52507a\"/\"sub-test-model-52507a-kserve-self-signed-certs\""
Jul 23 14:25:46.784036 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:46.783919    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-52507a\"/\"kube-root-ca.crt\""
Jul 23 14:25:46.784036 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:46.783958    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-52507a\"/\"default-dockercfg-rn8zk\""
Jul 23 14:25:46.793358 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:46.793335    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"]
Jul 23 14:25:46.904763 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:46.904727    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-dshm\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:46.904763 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:46.904765    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-model-cache\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:46.905052 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:46.904789    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-tmp-dir\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:46.905052 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:46.904884    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mrc2\" (UniqueName: \"kubernetes.io/projected/cbb712d4-4443-4fe5-88a3-4d289bf110a0-kube-api-access-2mrc2\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:46.905052 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:46.904928    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb712d4-4443-4fe5-88a3-4d289bf110a0-tls-certs\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:46.905052 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:46.904973    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-home\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:47.005918 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:47.005876    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb712d4-4443-4fe5-88a3-4d289bf110a0-tls-certs\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:47.006085 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:47.005936    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-home\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:47.006085 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:47.006034    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-dshm\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:47.006085 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:47.006059    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-model-cache\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:47.006085 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:47.006081    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-tmp-dir\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:47.006271 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:47.006124    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mrc2\" (UniqueName: \"kubernetes.io/projected/cbb712d4-4443-4fe5-88a3-4d289bf110a0-kube-api-access-2mrc2\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:47.006431 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:47.006405    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-home\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:47.006517 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:47.006416    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-model-cache\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:47.006584 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:47.006563    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-tmp-dir\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:47.008318 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:47.008290    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-dshm\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:47.008468 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:47.008452    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb712d4-4443-4fe5-88a3-4d289bf110a0-tls-certs\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:47.014997 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:47.014974    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mrc2\" (UniqueName: \"kubernetes.io/projected/cbb712d4-4443-4fe5-88a3-4d289bf110a0-kube-api-access-2mrc2\") pod \"sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") " pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:47.089954 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:47.089871    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:47.226581 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:47.226552    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"]
Jul 23 14:25:47.229032 ip-10-0-138-125 kubenswrapper[2568]: W0723 14:25:47.229001    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbb712d4_4443_4fe5_88a3_4d289bf110a0.slice/crio-5bd657ba81cdc8d5f7c882a11785e3fffe37d7f251ee9b7b0a648f6621539ef2 WatchSource:0}: Error finding container 5bd657ba81cdc8d5f7c882a11785e3fffe37d7f251ee9b7b0a648f6621539ef2: Status 404 returned error can't find the container with id 5bd657ba81cdc8d5f7c882a11785e3fffe37d7f251ee9b7b0a648f6621539ef2
Jul 23 14:25:48.053063 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:48.053027    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh" event={"ID":"cbb712d4-4443-4fe5-88a3-4d289bf110a0","Type":"ContainerStarted","Data":"81f027952fbc8be57ce0cb77c96a57dfc9aaa1d65b812605d01d4717eee6a804"}
Jul 23 14:25:48.053063 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:48.053068    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh" event={"ID":"cbb712d4-4443-4fe5-88a3-4d289bf110a0","Type":"ContainerStarted","Data":"5bd657ba81cdc8d5f7c882a11785e3fffe37d7f251ee9b7b0a648f6621539ef2"}
Jul 23 14:25:48.073273 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:48.073227    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh" podStartSLOduration=2.073211225 podStartE2EDuration="2.073211225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:25:48.071825784 +0000 UTC m=+2696.532916848" watchObservedRunningTime="2026-07-23 14:25:48.073211225 +0000 UTC m=+2696.534302301"
Jul 23 14:25:50.488477 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:50.488436    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:25:57.090175 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:57.090130    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:57.090175 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:57.090193    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:57.102737 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:57.102708    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:25:58.109990 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:25:58.109961    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:26:04.395081 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:04.395046    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:26:23.100327 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:23.100294    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:26:28.798106 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:28.798077    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"]
Jul 23 14:26:29.686180 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:29.686136    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"]
Jul 23 14:26:37.996730 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:37.996688    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"]
Jul 23 14:26:37.999580 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:37.999560    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.004174 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.004155    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-42e993\"/\"rate-test-model-42e993-kserve-self-signed-certs\""
Jul 23 14:26:38.017480 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.017457    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"]
Jul 23 14:26:38.162563 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.162530    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm9cn\" (UniqueName: \"kubernetes.io/projected/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-kube-api-access-lm9cn\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.162563 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.162567    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-home\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.162766 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.162636    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-tmp-dir\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.162766 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.162655    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-dshm\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.162766 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.162683    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-tls-certs\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.162907 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.162757    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-model-cache\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.263211 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.263127    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-tmp-dir\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.263211 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.263163    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-dshm\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.263211 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.263184    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-tls-certs\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.263474 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.263218    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-model-cache\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.263474 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.263292    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9cn\" (UniqueName: \"kubernetes.io/projected/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-kube-api-access-lm9cn\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.263474 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.263333    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-home\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.263725 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.263701    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-model-cache\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.263813 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.263721    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-home\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.263813 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.263728    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-tmp-dir\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.265823 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.265798    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-tls-certs\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.265823 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.265817    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-dshm\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.272404 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.272381    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9cn\" (UniqueName: \"kubernetes.io/projected/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-kube-api-access-lm9cn\") pod \"rate-test-model-42e993-kserve-7b9599c58c-ltmm7\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") " pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.308381 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.308351    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:38.443175 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:38.443152    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"]
Jul 23 14:26:38.445176 ip-10-0-138-125 kubenswrapper[2568]: W0723 14:26:38.445147    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2eb6589_2293_4f2a_b20d_d44c44bfaecc.slice/crio-0d5309c4f7cdf1eac10e5d6fe806954725005355657cff443e49b4cf717dc208 WatchSource:0}: Error finding container 0d5309c4f7cdf1eac10e5d6fe806954725005355657cff443e49b4cf717dc208: Status 404 returned error can't find the container with id 0d5309c4f7cdf1eac10e5d6fe806954725005355657cff443e49b4cf717dc208
Jul 23 14:26:39.285740 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:39.285708    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7" event={"ID":"a2eb6589-2293-4f2a-b20d-d44c44bfaecc","Type":"ContainerStarted","Data":"ab6263f81e3627152bebe96db2d3e7bfc3268bdfd29fdd2a687d936888f60b72"}
Jul 23 14:26:39.285740 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:39.285748    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7" event={"ID":"a2eb6589-2293-4f2a-b20d-d44c44bfaecc","Type":"ContainerStarted","Data":"0d5309c4f7cdf1eac10e5d6fe806954725005355657cff443e49b4cf717dc208"}
Jul 23 14:26:39.306923 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:39.306834    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7" podStartSLOduration=2.306817417 podStartE2EDuration="2.306817417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:26:39.306208801 +0000 UTC m=+2747.767299844" watchObservedRunningTime="2026-07-23 14:26:39.306817417 +0000 UTC m=+2747.767908453"
Jul 23 14:26:42.092287 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:42.092248    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:26:43.817587 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:43.817550    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7" podUID="6e78a980-2627-44e6-8e5e-efe403844bdd" containerName="main" containerID="cri-o://82f5e2ac883f810ee6b51129a5085228469d61f0e36e4fd97db19562d450a3cf" gracePeriod=45
Jul 23 14:26:44.072968 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.072873    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:26:44.220131 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.220093    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-home\") pod \"6e78a980-2627-44e6-8e5e-efe403844bdd\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") "
Jul 23 14:26:44.220328 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.220153    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bzhg\" (UniqueName: \"kubernetes.io/projected/6e78a980-2627-44e6-8e5e-efe403844bdd-kube-api-access-6bzhg\") pod \"6e78a980-2627-44e6-8e5e-efe403844bdd\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") "
Jul 23 14:26:44.220328 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.220183    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-dshm\") pod \"6e78a980-2627-44e6-8e5e-efe403844bdd\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") "
Jul 23 14:26:44.220328 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.220222    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e78a980-2627-44e6-8e5e-efe403844bdd-tls-certs\") pod \"6e78a980-2627-44e6-8e5e-efe403844bdd\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") "
Jul 23 14:26:44.220328 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.220275    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-tmp-dir\") pod \"6e78a980-2627-44e6-8e5e-efe403844bdd\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") "
Jul 23 14:26:44.220328 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.220297    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-model-cache\") pod \"6e78a980-2627-44e6-8e5e-efe403844bdd\" (UID: \"6e78a980-2627-44e6-8e5e-efe403844bdd\") "
Jul 23 14:26:44.220593 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.220403    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-home" (OuterVolumeSpecName: "home") pod "6e78a980-2627-44e6-8e5e-efe403844bdd" (UID: "6e78a980-2627-44e6-8e5e-efe403844bdd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:26:44.220653 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.220579    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e78a980-2627-44e6-8e5e-efe403844bdd" (UID: "6e78a980-2627-44e6-8e5e-efe403844bdd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:26:44.220653 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.220625    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e78a980-2627-44e6-8e5e-efe403844bdd" (UID: "6e78a980-2627-44e6-8e5e-efe403844bdd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:26:44.220653 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.220631    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-home\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:26:44.222349 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.222329    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-dshm" (OuterVolumeSpecName: "dshm") pod "6e78a980-2627-44e6-8e5e-efe403844bdd" (UID: "6e78a980-2627-44e6-8e5e-efe403844bdd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:26:44.222472 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.222444    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e78a980-2627-44e6-8e5e-efe403844bdd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e78a980-2627-44e6-8e5e-efe403844bdd" (UID: "6e78a980-2627-44e6-8e5e-efe403844bdd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:26:44.222529 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.222455    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e78a980-2627-44e6-8e5e-efe403844bdd-kube-api-access-6bzhg" (OuterVolumeSpecName: "kube-api-access-6bzhg") pod "6e78a980-2627-44e6-8e5e-efe403844bdd" (UID: "6e78a980-2627-44e6-8e5e-efe403844bdd"). InnerVolumeSpecName "kube-api-access-6bzhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:26:44.311671 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.311635    2568 generic.go:356] "Generic (PLEG): container finished" podID="6e78a980-2627-44e6-8e5e-efe403844bdd" containerID="82f5e2ac883f810ee6b51129a5085228469d61f0e36e4fd97db19562d450a3cf" exitCode=0
Jul 23 14:26:44.311835 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.311679    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"
Jul 23 14:26:44.311835 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.311724    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7" event={"ID":"6e78a980-2627-44e6-8e5e-efe403844bdd","Type":"ContainerDied","Data":"82f5e2ac883f810ee6b51129a5085228469d61f0e36e4fd97db19562d450a3cf"}
Jul 23 14:26:44.311835 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.311772    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7" event={"ID":"6e78a980-2627-44e6-8e5e-efe403844bdd","Type":"ContainerDied","Data":"35fd6c86ed840a10430950c709869f12350df9d5c5d604d519081cf7a9d736fe"}
Jul 23 14:26:44.311835 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.311796    2568 scope.go:117] "RemoveContainer" containerID="82f5e2ac883f810ee6b51129a5085228469d61f0e36e4fd97db19562d450a3cf"
Jul 23 14:26:44.321328 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.321311    2568 scope.go:117] "RemoveContainer" containerID="82f5e2ac883f810ee6b51129a5085228469d61f0e36e4fd97db19562d450a3cf"
Jul 23 14:26:44.321397 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.321310    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bzhg\" (UniqueName: \"kubernetes.io/projected/6e78a980-2627-44e6-8e5e-efe403844bdd-kube-api-access-6bzhg\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:26:44.321447 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.321410    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-dshm\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:26:44.321447 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.321428    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e78a980-2627-44e6-8e5e-efe403844bdd-tls-certs\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:26:44.321447 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.321441    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-tmp-dir\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:26:44.321558 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.321453    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e78a980-2627-44e6-8e5e-efe403844bdd-model-cache\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:26:44.321593 ip-10-0-138-125 kubenswrapper[2568]: E0723 14:26:44.321551    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f5e2ac883f810ee6b51129a5085228469d61f0e36e4fd97db19562d450a3cf\": container with ID starting with 82f5e2ac883f810ee6b51129a5085228469d61f0e36e4fd97db19562d450a3cf not found: ID does not exist" containerID="82f5e2ac883f810ee6b51129a5085228469d61f0e36e4fd97db19562d450a3cf"
Jul 23 14:26:44.321593 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.321572    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f5e2ac883f810ee6b51129a5085228469d61f0e36e4fd97db19562d450a3cf"} err="failed to get container status \"82f5e2ac883f810ee6b51129a5085228469d61f0e36e4fd97db19562d450a3cf\": rpc error: code = NotFound desc = could not find container \"82f5e2ac883f810ee6b51129a5085228469d61f0e36e4fd97db19562d450a3cf\": container with ID starting with 82f5e2ac883f810ee6b51129a5085228469d61f0e36e4fd97db19562d450a3cf not found: ID does not exist"
Jul 23 14:26:44.333983 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.333933    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"]
Jul 23 14:26:44.343816 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.339299    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-42e993/sub-test-model-42e993-kserve-866788d9dc-bvss7"]
Jul 23 14:26:44.706056 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.705968    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh" podUID="cbb712d4-4443-4fe5-88a3-4d289bf110a0" containerName="main" containerID="cri-o://81f027952fbc8be57ce0cb77c96a57dfc9aaa1d65b812605d01d4717eee6a804" gracePeriod=45
Jul 23 14:26:44.949769 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:44.949746    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:26:45.128237 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.128182    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-dshm\") pod \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") "
Jul 23 14:26:45.128237 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.128246    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb712d4-4443-4fe5-88a3-4d289bf110a0-tls-certs\") pod \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") "
Jul 23 14:26:45.128511 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.128288    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-home\") pod \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") "
Jul 23 14:26:45.128511 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.128327    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-model-cache\") pod \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") "
Jul 23 14:26:45.128511 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.128350    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-tmp-dir\") pod \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") "
Jul 23 14:26:45.128511 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.128465    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mrc2\" (UniqueName: \"kubernetes.io/projected/cbb712d4-4443-4fe5-88a3-4d289bf110a0-kube-api-access-2mrc2\") pod \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\" (UID: \"cbb712d4-4443-4fe5-88a3-4d289bf110a0\") "
Jul 23 14:26:45.128722 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.128594    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-home" (OuterVolumeSpecName: "home") pod "cbb712d4-4443-4fe5-88a3-4d289bf110a0" (UID: "cbb712d4-4443-4fe5-88a3-4d289bf110a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:26:45.128722 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.128613    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "cbb712d4-4443-4fe5-88a3-4d289bf110a0" (UID: "cbb712d4-4443-4fe5-88a3-4d289bf110a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:26:45.128722 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.128664    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cbb712d4-4443-4fe5-88a3-4d289bf110a0" (UID: "cbb712d4-4443-4fe5-88a3-4d289bf110a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:26:45.128895 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.128837    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-model-cache\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:26:45.128961 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.128905    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-tmp-dir\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:26:45.128961 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.128915    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-home\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:26:45.130619 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.130586    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb712d4-4443-4fe5-88a3-4d289bf110a0-kube-api-access-2mrc2" (OuterVolumeSpecName: "kube-api-access-2mrc2") pod "cbb712d4-4443-4fe5-88a3-4d289bf110a0" (UID: "cbb712d4-4443-4fe5-88a3-4d289bf110a0"). InnerVolumeSpecName "kube-api-access-2mrc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:26:45.130883 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.130863    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-dshm" (OuterVolumeSpecName: "dshm") pod "cbb712d4-4443-4fe5-88a3-4d289bf110a0" (UID: "cbb712d4-4443-4fe5-88a3-4d289bf110a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:26:45.130961 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.130884    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbb712d4-4443-4fe5-88a3-4d289bf110a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cbb712d4-4443-4fe5-88a3-4d289bf110a0" (UID: "cbb712d4-4443-4fe5-88a3-4d289bf110a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:26:45.230188 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.230157    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mrc2\" (UniqueName: \"kubernetes.io/projected/cbb712d4-4443-4fe5-88a3-4d289bf110a0-kube-api-access-2mrc2\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:26:45.230188 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.230184    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb712d4-4443-4fe5-88a3-4d289bf110a0-dshm\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:26:45.230188 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.230195    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb712d4-4443-4fe5-88a3-4d289bf110a0-tls-certs\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:26:45.317948 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.317919    2568 generic.go:356] "Generic (PLEG): container finished" podID="cbb712d4-4443-4fe5-88a3-4d289bf110a0" containerID="81f027952fbc8be57ce0cb77c96a57dfc9aaa1d65b812605d01d4717eee6a804" exitCode=0
Jul 23 14:26:45.318087 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.317966    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"
Jul 23 14:26:45.318087 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.317981    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh" event={"ID":"cbb712d4-4443-4fe5-88a3-4d289bf110a0","Type":"ContainerDied","Data":"81f027952fbc8be57ce0cb77c96a57dfc9aaa1d65b812605d01d4717eee6a804"}
Jul 23 14:26:45.318087 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.318010    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh" event={"ID":"cbb712d4-4443-4fe5-88a3-4d289bf110a0","Type":"ContainerDied","Data":"5bd657ba81cdc8d5f7c882a11785e3fffe37d7f251ee9b7b0a648f6621539ef2"}
Jul 23 14:26:45.318087 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.318029    2568 scope.go:117] "RemoveContainer" containerID="81f027952fbc8be57ce0cb77c96a57dfc9aaa1d65b812605d01d4717eee6a804"
Jul 23 14:26:45.327042 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.327026    2568 scope.go:117] "RemoveContainer" containerID="81f027952fbc8be57ce0cb77c96a57dfc9aaa1d65b812605d01d4717eee6a804"
Jul 23 14:26:45.327309 ip-10-0-138-125 kubenswrapper[2568]: E0723 14:26:45.327291    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f027952fbc8be57ce0cb77c96a57dfc9aaa1d65b812605d01d4717eee6a804\": container with ID starting with 81f027952fbc8be57ce0cb77c96a57dfc9aaa1d65b812605d01d4717eee6a804 not found: ID does not exist" containerID="81f027952fbc8be57ce0cb77c96a57dfc9aaa1d65b812605d01d4717eee6a804"
Jul 23 14:26:45.327373 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.327321    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f027952fbc8be57ce0cb77c96a57dfc9aaa1d65b812605d01d4717eee6a804"} err="failed to get container status \"81f027952fbc8be57ce0cb77c96a57dfc9aaa1d65b812605d01d4717eee6a804\": rpc error: code = NotFound desc = could not find container \"81f027952fbc8be57ce0cb77c96a57dfc9aaa1d65b812605d01d4717eee6a804\": container with ID starting with 81f027952fbc8be57ce0cb77c96a57dfc9aaa1d65b812605d01d4717eee6a804 not found: ID does not exist"
Jul 23 14:26:45.343864 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.343828    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"]
Jul 23 14:26:45.345344 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:45.345324    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-52507a/sub-test-model-52507a-kserve-79fd7bf8c5-gr7vh"]
Jul 23 14:26:46.152418 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:46.152384    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e78a980-2627-44e6-8e5e-efe403844bdd" path="/var/lib/kubelet/pods/6e78a980-2627-44e6-8e5e-efe403844bdd/volumes"
Jul 23 14:26:46.152783 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:46.152762    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbb712d4-4443-4fe5-88a3-4d289bf110a0" path="/var/lib/kubelet/pods/cbb712d4-4443-4fe5-88a3-4d289bf110a0/volumes"
Jul 23 14:26:48.309335 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:48.309299    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:48.309772 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:48.309487    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:48.321975 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:48.321951    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:26:48.345602 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:26:48.345573    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:27:02.604396 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.604350    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"]
Jul 23 14:27:02.604800 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.604755    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb712d4-4443-4fe5-88a3-4d289bf110a0" containerName="main"
Jul 23 14:27:02.604800 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.604771    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb712d4-4443-4fe5-88a3-4d289bf110a0" containerName="main"
Jul 23 14:27:02.604800 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.604798    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e78a980-2627-44e6-8e5e-efe403844bdd" containerName="main"
Jul 23 14:27:02.604929 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.604808    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e78a980-2627-44e6-8e5e-efe403844bdd" containerName="main"
Jul 23 14:27:02.604929 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.604917    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb712d4-4443-4fe5-88a3-4d289bf110a0" containerName="main"
Jul 23 14:27:02.604999 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.604941    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e78a980-2627-44e6-8e5e-efe403844bdd" containerName="main"
Jul 23 14:27:02.607970 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.607954    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.611752 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.611725    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-52507a\"/\"kube-root-ca.crt\""
Jul 23 14:27:02.611906 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.611725    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-52507a\"/\"rate-test-model-52507a-kserve-self-signed-certs\""
Jul 23 14:27:02.611906 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.611728    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-52507a\"/\"default-dockercfg-rn8zk\""
Jul 23 14:27:02.611906 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.611829    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-52507a\"/\"openshift-service-ca.crt\""
Jul 23 14:27:02.620782 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.620762    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"]
Jul 23 14:27:02.659738 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.659704    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-dshm\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.659912 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.659781    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-tmp-dir\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.659912 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.659840    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1983b7-7214-4fae-9157-b989ff625057-tls-certs\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.660254 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.660232    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-home\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.661466 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.661434    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8tpd\" (UniqueName: \"kubernetes.io/projected/2a1983b7-7214-4fae-9157-b989ff625057-kube-api-access-g8tpd\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.661559 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.661496    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-model-cache\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.761879 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.761841    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1983b7-7214-4fae-9157-b989ff625057-tls-certs\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.762000 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.761914    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-home\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.762000 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.761936    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8tpd\" (UniqueName: \"kubernetes.io/projected/2a1983b7-7214-4fae-9157-b989ff625057-kube-api-access-g8tpd\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.762000 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.761963    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-model-cache\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.762000 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.761993    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-dshm\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.762310 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.762014    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-tmp-dir\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.762436 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.762408    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-home\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.762515 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.762471    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-model-cache\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.762575 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.762540    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-tmp-dir\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.764391 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.764369    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-dshm\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.764682 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.764665    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1983b7-7214-4fae-9157-b989ff625057-tls-certs\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.771052 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.771034    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8tpd\" (UniqueName: \"kubernetes.io/projected/2a1983b7-7214-4fae-9157-b989ff625057-kube-api-access-g8tpd\") pod \"rate-test-model-52507a-kserve-869fd4c8dc-jm75m\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") " pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:02.916613 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:02.916513    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:03.054652 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:03.054624    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"]
Jul 23 14:27:03.056215 ip-10-0-138-125 kubenswrapper[2568]: W0723 14:27:03.056191    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a1983b7_7214_4fae_9157_b989ff625057.slice/crio-61bfa24273dd8747ce73b3751cd45a95bc3de1ff8bfa4504afc4b8e24570f098 WatchSource:0}: Error finding container 61bfa24273dd8747ce73b3751cd45a95bc3de1ff8bfa4504afc4b8e24570f098: Status 404 returned error can't find the container with id 61bfa24273dd8747ce73b3751cd45a95bc3de1ff8bfa4504afc4b8e24570f098
Jul 23 14:27:03.404719 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:03.404679    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m" event={"ID":"2a1983b7-7214-4fae-9157-b989ff625057","Type":"ContainerStarted","Data":"b9184c689736b462a503c77b460f69ada60c87d519baa2f9d7866f6aeef802c3"}
Jul 23 14:27:03.404915 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:03.404727    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m" event={"ID":"2a1983b7-7214-4fae-9157-b989ff625057","Type":"ContainerStarted","Data":"61bfa24273dd8747ce73b3751cd45a95bc3de1ff8bfa4504afc4b8e24570f098"}
Jul 23 14:27:03.424990 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:03.424928    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m" podStartSLOduration=1.424905744 podStartE2EDuration="1.424905744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 14:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:27:03.423503159 +0000 UTC m=+2771.884594202" watchObservedRunningTime="2026-07-23 14:27:03.424905744 +0000 UTC m=+2771.885996786"
Jul 23 14:27:06.209750 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:06.209718    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:27:12.917658 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:12.917615    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:12.920187 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:12.917677    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:12.930348 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:12.930324    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:13.457746 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:13.457720    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:27:20.939772 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:20.939728    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:27:37.291675 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:37.291644    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:27:43.976626 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:43.976589    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"]
Jul 23 14:27:44.821210 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:44.821174    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"]
Jul 23 14:27:54.994847 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:54.994808    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:27:58.996395 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:58.996335    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7" podUID="a2eb6589-2293-4f2a-b20d-d44c44bfaecc" containerName="main" containerID="cri-o://ab6263f81e3627152bebe96db2d3e7bfc3268bdfd29fdd2a687d936888f60b72" gracePeriod=45
Jul 23 14:27:59.239332 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.239301    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:27:59.339914 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.339828    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-dshm\") pod \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") "
Jul 23 14:27:59.339914 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.339906    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-tls-certs\") pod \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") "
Jul 23 14:27:59.340094 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.339926    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-tmp-dir\") pod \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") "
Jul 23 14:27:59.340094 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.339941    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-model-cache\") pod \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") "
Jul 23 14:27:59.340094 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.339967    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-home\") pod \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") "
Jul 23 14:27:59.340094 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.339988    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm9cn\" (UniqueName: \"kubernetes.io/projected/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-kube-api-access-lm9cn\") pod \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\" (UID: \"a2eb6589-2293-4f2a-b20d-d44c44bfaecc\") "
Jul 23 14:27:59.340305 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.340208    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2eb6589-2293-4f2a-b20d-d44c44bfaecc" (UID: "a2eb6589-2293-4f2a-b20d-d44c44bfaecc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:27:59.340305 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.340193    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2eb6589-2293-4f2a-b20d-d44c44bfaecc" (UID: "a2eb6589-2293-4f2a-b20d-d44c44bfaecc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:27:59.340305 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.340261    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-home" (OuterVolumeSpecName: "home") pod "a2eb6589-2293-4f2a-b20d-d44c44bfaecc" (UID: "a2eb6589-2293-4f2a-b20d-d44c44bfaecc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:27:59.342008 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.341984    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-dshm" (OuterVolumeSpecName: "dshm") pod "a2eb6589-2293-4f2a-b20d-d44c44bfaecc" (UID: "a2eb6589-2293-4f2a-b20d-d44c44bfaecc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:27:59.342110 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.342023    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-kube-api-access-lm9cn" (OuterVolumeSpecName: "kube-api-access-lm9cn") pod "a2eb6589-2293-4f2a-b20d-d44c44bfaecc" (UID: "a2eb6589-2293-4f2a-b20d-d44c44bfaecc"). InnerVolumeSpecName "kube-api-access-lm9cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:27:59.342110 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.342101    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2eb6589-2293-4f2a-b20d-d44c44bfaecc" (UID: "a2eb6589-2293-4f2a-b20d-d44c44bfaecc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:27:59.441519 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.441494    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-tls-certs\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:27:59.441519 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.441515    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-tmp-dir\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:27:59.441519 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.441523    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-model-cache\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:27:59.441702 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.441532    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-home\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:27:59.441702 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.441539    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm9cn\" (UniqueName: \"kubernetes.io/projected/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-kube-api-access-lm9cn\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:27:59.441702 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.441549    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2eb6589-2293-4f2a-b20d-d44c44bfaecc-dshm\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:27:59.660736 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.660646    2568 generic.go:356] "Generic (PLEG): container finished" podID="a2eb6589-2293-4f2a-b20d-d44c44bfaecc" containerID="ab6263f81e3627152bebe96db2d3e7bfc3268bdfd29fdd2a687d936888f60b72" exitCode=0
Jul 23 14:27:59.660736 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.660691    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"
Jul 23 14:27:59.660736 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.660698    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7" event={"ID":"a2eb6589-2293-4f2a-b20d-d44c44bfaecc","Type":"ContainerDied","Data":"ab6263f81e3627152bebe96db2d3e7bfc3268bdfd29fdd2a687d936888f60b72"}
Jul 23 14:27:59.660736 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.660730    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7" event={"ID":"a2eb6589-2293-4f2a-b20d-d44c44bfaecc","Type":"ContainerDied","Data":"0d5309c4f7cdf1eac10e5d6fe806954725005355657cff443e49b4cf717dc208"}
Jul 23 14:27:59.661118 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.660746    2568 scope.go:117] "RemoveContainer" containerID="ab6263f81e3627152bebe96db2d3e7bfc3268bdfd29fdd2a687d936888f60b72"
Jul 23 14:27:59.669987 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.669964    2568 scope.go:117] "RemoveContainer" containerID="ab6263f81e3627152bebe96db2d3e7bfc3268bdfd29fdd2a687d936888f60b72"
Jul 23 14:27:59.670228 ip-10-0-138-125 kubenswrapper[2568]: E0723 14:27:59.670211    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6263f81e3627152bebe96db2d3e7bfc3268bdfd29fdd2a687d936888f60b72\": container with ID starting with ab6263f81e3627152bebe96db2d3e7bfc3268bdfd29fdd2a687d936888f60b72 not found: ID does not exist" containerID="ab6263f81e3627152bebe96db2d3e7bfc3268bdfd29fdd2a687d936888f60b72"
Jul 23 14:27:59.670277 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.670235    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6263f81e3627152bebe96db2d3e7bfc3268bdfd29fdd2a687d936888f60b72"} err="failed to get container status \"ab6263f81e3627152bebe96db2d3e7bfc3268bdfd29fdd2a687d936888f60b72\": rpc error: code = NotFound desc = could not find container \"ab6263f81e3627152bebe96db2d3e7bfc3268bdfd29fdd2a687d936888f60b72\": container with ID starting with ab6263f81e3627152bebe96db2d3e7bfc3268bdfd29fdd2a687d936888f60b72 not found: ID does not exist"
Jul 23 14:27:59.690910 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.690886    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"]
Jul 23 14:27:59.699617 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.699596    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-42e993/rate-test-model-42e993-kserve-7b9599c58c-ltmm7"]
Jul 23 14:27:59.840675 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:27:59.840642    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m" podUID="2a1983b7-7214-4fae-9157-b989ff625057" containerName="main" containerID="cri-o://b9184c689736b462a503c77b460f69ada60c87d519baa2f9d7866f6aeef802c3" gracePeriod=45
Jul 23 14:28:00.088652 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.088628    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:28:00.147339 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.147315    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-tmp-dir\") pod \"2a1983b7-7214-4fae-9157-b989ff625057\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") "
Jul 23 14:28:00.147339 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.147342    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-home\") pod \"2a1983b7-7214-4fae-9157-b989ff625057\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") "
Jul 23 14:28:00.147568 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.147361    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-model-cache\") pod \"2a1983b7-7214-4fae-9157-b989ff625057\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") "
Jul 23 14:28:00.147568 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.147436    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1983b7-7214-4fae-9157-b989ff625057-tls-certs\") pod \"2a1983b7-7214-4fae-9157-b989ff625057\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") "
Jul 23 14:28:00.147568 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.147456    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-dshm\") pod \"2a1983b7-7214-4fae-9157-b989ff625057\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") "
Jul 23 14:28:00.147568 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.147489    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8tpd\" (UniqueName: \"kubernetes.io/projected/2a1983b7-7214-4fae-9157-b989ff625057-kube-api-access-g8tpd\") pod \"2a1983b7-7214-4fae-9157-b989ff625057\" (UID: \"2a1983b7-7214-4fae-9157-b989ff625057\") "
Jul 23 14:28:00.147765 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.147571    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a1983b7-7214-4fae-9157-b989ff625057" (UID: "2a1983b7-7214-4fae-9157-b989ff625057"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:28:00.147765 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.147649    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-home" (OuterVolumeSpecName: "home") pod "2a1983b7-7214-4fae-9157-b989ff625057" (UID: "2a1983b7-7214-4fae-9157-b989ff625057"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:28:00.147765 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.147705    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a1983b7-7214-4fae-9157-b989ff625057" (UID: "2a1983b7-7214-4fae-9157-b989ff625057"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:28:00.147901 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.147822    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-tmp-dir\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:28:00.147901 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.147842    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-home\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:28:00.147901 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.147871    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-model-cache\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:28:00.149601 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.149559    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-dshm" (OuterVolumeSpecName: "dshm") pod "2a1983b7-7214-4fae-9157-b989ff625057" (UID: "2a1983b7-7214-4fae-9157-b989ff625057"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 14:28:00.149601 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.149573    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1983b7-7214-4fae-9157-b989ff625057-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a1983b7-7214-4fae-9157-b989ff625057" (UID: "2a1983b7-7214-4fae-9157-b989ff625057"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 14:28:00.149893 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.149873    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1983b7-7214-4fae-9157-b989ff625057-kube-api-access-g8tpd" (OuterVolumeSpecName: "kube-api-access-g8tpd") pod "2a1983b7-7214-4fae-9157-b989ff625057" (UID: "2a1983b7-7214-4fae-9157-b989ff625057"). InnerVolumeSpecName "kube-api-access-g8tpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:28:00.152597 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.152574    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2eb6589-2293-4f2a-b20d-d44c44bfaecc" path="/var/lib/kubelet/pods/a2eb6589-2293-4f2a-b20d-d44c44bfaecc/volumes"
Jul 23 14:28:00.248475 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.248429    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1983b7-7214-4fae-9157-b989ff625057-tls-certs\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:28:00.248475 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.248474    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1983b7-7214-4fae-9157-b989ff625057-dshm\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:28:00.248638 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.248483    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8tpd\" (UniqueName: \"kubernetes.io/projected/2a1983b7-7214-4fae-9157-b989ff625057-kube-api-access-g8tpd\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:28:00.666846 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.666755    2568 generic.go:356] "Generic (PLEG): container finished" podID="2a1983b7-7214-4fae-9157-b989ff625057" containerID="b9184c689736b462a503c77b460f69ada60c87d519baa2f9d7866f6aeef802c3" exitCode=0
Jul 23 14:28:00.666846 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.666799    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"
Jul 23 14:28:00.667105 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.666848    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m" event={"ID":"2a1983b7-7214-4fae-9157-b989ff625057","Type":"ContainerDied","Data":"b9184c689736b462a503c77b460f69ada60c87d519baa2f9d7866f6aeef802c3"}
Jul 23 14:28:00.667105 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.666918    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m" event={"ID":"2a1983b7-7214-4fae-9157-b989ff625057","Type":"ContainerDied","Data":"61bfa24273dd8747ce73b3751cd45a95bc3de1ff8bfa4504afc4b8e24570f098"}
Jul 23 14:28:00.667105 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.666941    2568 scope.go:117] "RemoveContainer" containerID="b9184c689736b462a503c77b460f69ada60c87d519baa2f9d7866f6aeef802c3"
Jul 23 14:28:00.675271 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.675253    2568 scope.go:117] "RemoveContainer" containerID="b9184c689736b462a503c77b460f69ada60c87d519baa2f9d7866f6aeef802c3"
Jul 23 14:28:00.675512 ip-10-0-138-125 kubenswrapper[2568]: E0723 14:28:00.675494    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9184c689736b462a503c77b460f69ada60c87d519baa2f9d7866f6aeef802c3\": container with ID starting with b9184c689736b462a503c77b460f69ada60c87d519baa2f9d7866f6aeef802c3 not found: ID does not exist" containerID="b9184c689736b462a503c77b460f69ada60c87d519baa2f9d7866f6aeef802c3"
Jul 23 14:28:00.675562 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.675520    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9184c689736b462a503c77b460f69ada60c87d519baa2f9d7866f6aeef802c3"} err="failed to get container status \"b9184c689736b462a503c77b460f69ada60c87d519baa2f9d7866f6aeef802c3\": rpc error: code = NotFound desc = could not find container \"b9184c689736b462a503c77b460f69ada60c87d519baa2f9d7866f6aeef802c3\": container with ID starting with b9184c689736b462a503c77b460f69ada60c87d519baa2f9d7866f6aeef802c3 not found: ID does not exist"
Jul 23 14:28:00.684494 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.684472    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"]
Jul 23 14:28:00.691181 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.691159    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-52507a/rate-test-model-52507a-kserve-869fd4c8dc-jm75m"]
Jul 23 14:28:00.902177 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.902142    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7630-140098424682512"]
Jul 23 14:28:00.902463 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.902452    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1983b7-7214-4fae-9157-b989ff625057" containerName="main"
Jul 23 14:28:00.902508 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.902465    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1983b7-7214-4fae-9157-b989ff625057" containerName="main"
Jul 23 14:28:00.902508 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.902477    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2eb6589-2293-4f2a-b20d-d44c44bfaecc" containerName="main"
Jul 23 14:28:00.902508 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.902482    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2eb6589-2293-4f2a-b20d-d44c44bfaecc" containerName="main"
Jul 23 14:28:00.902597 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.902554    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2eb6589-2293-4f2a-b20d-d44c44bfaecc" containerName="main"
Jul 23 14:28:00.902597 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.902561    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1983b7-7214-4fae-9157-b989ff625057" containerName="main"
Jul 23 14:28:00.906780 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.906760    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7630-140098424682512"
Jul 23 14:28:00.909302 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.909285    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-g7rfd\""
Jul 23 14:28:00.914520 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.914454    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7630-140098424682512"]
Jul 23 14:28:00.953686 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:00.953627    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpdv5\" (UniqueName: \"kubernetes.io/projected/ebf38fdf-6a25-4009-9d9b-3a72b19f3218-kube-api-access-wpdv5\") pod \"test-curl-7630-140098424682512\" (UID: \"ebf38fdf-6a25-4009-9d9b-3a72b19f3218\") " pod="opendatahub/test-curl-7630-140098424682512"
Jul 23 14:28:01.054986 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:01.054949    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpdv5\" (UniqueName: \"kubernetes.io/projected/ebf38fdf-6a25-4009-9d9b-3a72b19f3218-kube-api-access-wpdv5\") pod \"test-curl-7630-140098424682512\" (UID: \"ebf38fdf-6a25-4009-9d9b-3a72b19f3218\") " pod="opendatahub/test-curl-7630-140098424682512"
Jul 23 14:28:01.064080 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:01.064053    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpdv5\" (UniqueName: \"kubernetes.io/projected/ebf38fdf-6a25-4009-9d9b-3a72b19f3218-kube-api-access-wpdv5\") pod \"test-curl-7630-140098424682512\" (UID: \"ebf38fdf-6a25-4009-9d9b-3a72b19f3218\") " pod="opendatahub/test-curl-7630-140098424682512"
Jul 23 14:28:01.217490 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:01.217399    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7630-140098424682512"
Jul 23 14:28:01.351516 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:01.351486    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7630-140098424682512"]
Jul 23 14:28:01.354471 ip-10-0-138-125 kubenswrapper[2568]: W0723 14:28:01.354441    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebf38fdf_6a25_4009_9d9b_3a72b19f3218.slice/crio-34c60efd94664d7f52f1dfa3286c6af14e32c9fcf6b5ce403dd2d1dab99756a2 WatchSource:0}: Error finding container 34c60efd94664d7f52f1dfa3286c6af14e32c9fcf6b5ce403dd2d1dab99756a2: Status 404 returned error can't find the container with id 34c60efd94664d7f52f1dfa3286c6af14e32c9fcf6b5ce403dd2d1dab99756a2
Jul 23 14:28:01.355713 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:01.355694    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 14:28:01.672638 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:01.672606    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7630-140098424682512" event={"ID":"ebf38fdf-6a25-4009-9d9b-3a72b19f3218","Type":"ContainerStarted","Data":"34c60efd94664d7f52f1dfa3286c6af14e32c9fcf6b5ce403dd2d1dab99756a2"}
Jul 23 14:28:02.152329 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:02.152298    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a1983b7-7214-4fae-9157-b989ff625057" path="/var/lib/kubelet/pods/2a1983b7-7214-4fae-9157-b989ff625057/volumes"
Jul 23 14:28:02.678960 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:02.678929    2568 generic.go:356] "Generic (PLEG): container finished" podID="ebf38fdf-6a25-4009-9d9b-3a72b19f3218" containerID="e9af214ae80682fd3466e59745709294815fea67cc868049157c13aa50e8455b" exitCode=0
Jul 23 14:28:02.679344 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:02.679020    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7630-140098424682512" event={"ID":"ebf38fdf-6a25-4009-9d9b-3a72b19f3218","Type":"ContainerDied","Data":"e9af214ae80682fd3466e59745709294815fea67cc868049157c13aa50e8455b"}
Jul 23 14:28:03.814618 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:03.814590    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7630-140098424682512"
Jul 23 14:28:03.876613 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:03.876587    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7630-140098424682512_ebf38fdf-6a25-4009-9d9b-3a72b19f3218/test-curl-7630-140098424682512/0.log"
Jul 23 14:28:03.878789 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:03.878765    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpdv5\" (UniqueName: \"kubernetes.io/projected/ebf38fdf-6a25-4009-9d9b-3a72b19f3218-kube-api-access-wpdv5\") pod \"ebf38fdf-6a25-4009-9d9b-3a72b19f3218\" (UID: \"ebf38fdf-6a25-4009-9d9b-3a72b19f3218\") "
Jul 23 14:28:03.880835 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:03.880812    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf38fdf-6a25-4009-9d9b-3a72b19f3218-kube-api-access-wpdv5" (OuterVolumeSpecName: "kube-api-access-wpdv5") pod "ebf38fdf-6a25-4009-9d9b-3a72b19f3218" (UID: "ebf38fdf-6a25-4009-9d9b-3a72b19f3218"). InnerVolumeSpecName "kube-api-access-wpdv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 14:28:03.908418 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:03.908381    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7630-140098424682512"]
Jul 23 14:28:03.910684 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:03.910661    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7630-140098424682512"]
Jul 23 14:28:03.985109 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:03.985031    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpdv5\" (UniqueName: \"kubernetes.io/projected/ebf38fdf-6a25-4009-9d9b-3a72b19f3218-kube-api-access-wpdv5\") on node \"ip-10-0-138-125.ec2.internal\" DevicePath \"\""
Jul 23 14:28:04.151830 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:04.151798    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebf38fdf-6a25-4009-9d9b-3a72b19f3218" path="/var/lib/kubelet/pods/ebf38fdf-6a25-4009-9d9b-3a72b19f3218/volumes"
Jul 23 14:28:04.690601 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:04.690572    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7630-140098424682512"
Jul 23 14:28:04.690803 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:04.690608    2568 scope.go:117] "RemoveContainer" containerID="e9af214ae80682fd3466e59745709294815fea67cc868049157c13aa50e8455b"
Jul 23 14:28:17.786409 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:17.786375    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:28:40.086233 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:40.086198    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pbbwv"]
Jul 23 14:28:49.815732 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:49.815645    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f975f7564-rqn6c_e8a6c465-0bb9-42d9-a063-bc415edb082e/authorino/0.log"
Jul 23 14:28:57.066302 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:57.066267    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c8b88fb69-4snf8_78a34ac0-ed10-48a3-ab9e-deb9e54c8989/manager/0.log"
Jul 23 14:28:57.078116 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:57.078094    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c8b88fb69-4snf8_78a34ac0-ed10-48a3-ab9e-deb9e54c8989/copy-manifests/0.log"
Jul 23 14:28:59.541062 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:59.541031    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f975f7564-rqn6c_e8a6c465-0bb9-42d9-a063-bc415edb082e/authorino/0.log"
Jul 23 14:28:59.950663 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:28:59.950589    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-q6s6m_9bbbe4b6-d958-4723-80b9-7c91c4fa5246/kuadrant-console-plugin/0.log"
Jul 23 14:29:00.387373 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:00.387344    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-pbbwv_6aed8257-e716-4ec6-94ad-07232fffe314/limitador/0.log"
Jul 23 14:29:01.371246 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:01.371219    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d57cbb598-cfjg4_ca94efb8-d44b-43d6-8d89-ef09eb258bf8/kube-auth-proxy/0.log"
Jul 23 14:29:02.689246 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:02.689213    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5rkrf_581a4447-63d8-4008-8dd0-113e2c1598ca/main/0.log"
Jul 23 14:29:02.946503 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:02.946383    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-rpgvf_020e8ee6-58f1-400f-9649-f9e2aff96419/main/0.log"
Jul 23 14:29:12.874017 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:12.873982    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fvtk6_d6f1fe9a-e7d6-4f58-b01e-e821f8e945b1/global-pull-secret-syncer/0.log"
Jul 23 14:29:13.339488 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:13.339456    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xvlj5_932faad9-0c93-47e5-a018-b2b4e79aba10/konnectivity-agent/0.log"
Jul 23 14:29:13.444704 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:13.444671    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-125.ec2.internal_3de01a743a50264260f9a41b54c7e4a3/haproxy/0.log"
Jul 23 14:29:18.496944 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:18.496864    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f975f7564-rqn6c_e8a6c465-0bb9-42d9-a063-bc415edb082e/authorino/0.log"
Jul 23 14:29:18.631745 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:18.631716    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-q6s6m_9bbbe4b6-d958-4723-80b9-7c91c4fa5246/kuadrant-console-plugin/0.log"
Jul 23 14:29:18.938084 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:18.937998    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-pbbwv_6aed8257-e716-4ec6-94ad-07232fffe314/limitador/0.log"
Jul 23 14:29:20.716744 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:20.716711    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_64df05b4-0f98-4400-be4c-9cc80efcdfef/alertmanager/0.log"
Jul 23 14:29:20.800871 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:20.800824    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_64df05b4-0f98-4400-be4c-9cc80efcdfef/config-reloader/0.log"
Jul 23 14:29:20.853754 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:20.853726    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_64df05b4-0f98-4400-be4c-9cc80efcdfef/kube-rbac-proxy-web/0.log"
Jul 23 14:29:20.888204 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:20.888165    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_64df05b4-0f98-4400-be4c-9cc80efcdfef/kube-rbac-proxy/0.log"
Jul 23 14:29:20.926101 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:20.926072    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_64df05b4-0f98-4400-be4c-9cc80efcdfef/kube-rbac-proxy-metric/0.log"
Jul 23 14:29:20.961932 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:20.961906    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_64df05b4-0f98-4400-be4c-9cc80efcdfef/prom-label-proxy/0.log"
Jul 23 14:29:20.994028 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:20.993989    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_64df05b4-0f98-4400-be4c-9cc80efcdfef/init-config-reloader/0.log"
Jul 23 14:29:21.087125 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:21.087080    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-mlvrs_16adce2b-eee7-4f2a-9041-32ff625ba0ec/kube-state-metrics/0.log"
Jul 23 14:29:21.107097 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:21.107071    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-mlvrs_16adce2b-eee7-4f2a-9041-32ff625ba0ec/kube-rbac-proxy-main/0.log"
Jul 23 14:29:21.129001 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:21.128968    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-mlvrs_16adce2b-eee7-4f2a-9041-32ff625ba0ec/kube-rbac-proxy-self/0.log"
Jul 23 14:29:21.187383 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:21.187350    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-cw7mz_b70ebbe6-05ea-4095-bb0e-e39c2709ea86/monitoring-plugin/0.log"
Jul 23 14:29:21.244446 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:21.244365    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8ngmk_5e59d57f-280a-42f0-bd4d-15eedfefcdf9/node-exporter/0.log"
Jul 23 14:29:21.276361 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:21.276335    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8ngmk_5e59d57f-280a-42f0-bd4d-15eedfefcdf9/kube-rbac-proxy/0.log"
Jul 23 14:29:21.308472 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:21.308449    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8ngmk_5e59d57f-280a-42f0-bd4d-15eedfefcdf9/init-textfile/0.log"
Jul 23 14:29:21.525257 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:21.525176    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-m7swl_83ec07d6-e691-40d8-a261-9194df7ef9da/kube-rbac-proxy-main/0.log"
Jul 23 14:29:21.548478 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:21.548457    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-m7swl_83ec07d6-e691-40d8-a261-9194df7ef9da/kube-rbac-proxy-self/0.log"
Jul 23 14:29:21.573787 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:21.573763    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-m7swl_83ec07d6-e691-40d8-a261-9194df7ef9da/openshift-state-metrics/0.log"
Jul 23 14:29:21.986290 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:21.986204    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-rwml7_e48d72cd-4b5b-4514-9eef-0c599c5ee29f/prometheus-operator-admission-webhook/0.log"
Jul 23 14:29:22.030140 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.030117    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76bff58957-wwhfc_7d62d918-599a-438a-ac1f-2ba20a97a4f5/telemeter-client/0.log"
Jul 23 14:29:22.059360 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.059335    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76bff58957-wwhfc_7d62d918-599a-438a-ac1f-2ba20a97a4f5/reload/0.log"
Jul 23 14:29:22.083468 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.083444    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76bff58957-wwhfc_7d62d918-599a-438a-ac1f-2ba20a97a4f5/kube-rbac-proxy/0.log"
Jul 23 14:29:22.629636 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.629603    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"]
Jul 23 14:29:22.629985 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.629967    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf38fdf-6a25-4009-9d9b-3a72b19f3218" containerName="test-curl-7630-140098424682512"
Jul 23 14:29:22.630037 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.629986    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf38fdf-6a25-4009-9d9b-3a72b19f3218" containerName="test-curl-7630-140098424682512"
Jul 23 14:29:22.630073 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.630061    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf38fdf-6a25-4009-9d9b-3a72b19f3218" containerName="test-curl-7630-140098424682512"
Jul 23 14:29:22.632487 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.632471    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:22.640874 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.640827    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jrxnq\"/\"kube-root-ca.crt\""
Jul 23 14:29:22.641685 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.641667    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jrxnq\"/\"default-dockercfg-7h9v2\""
Jul 23 14:29:22.641783 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.641702    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jrxnq\"/\"openshift-service-ca.crt\""
Jul 23 14:29:22.658731 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.658707    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"]
Jul 23 14:29:22.730750 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.730717    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e9c9fa47-a912-49cc-8c2f-bf6a60df68a5-podres\") pod \"perf-node-gather-daemonset-zmdkm\" (UID: \"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5\") " pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:22.730926 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.730778    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9c9fa47-a912-49cc-8c2f-bf6a60df68a5-lib-modules\") pod \"perf-node-gather-daemonset-zmdkm\" (UID: \"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5\") " pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:22.730926 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.730802    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fcmj\" (UniqueName: \"kubernetes.io/projected/e9c9fa47-a912-49cc-8c2f-bf6a60df68a5-kube-api-access-8fcmj\") pod \"perf-node-gather-daemonset-zmdkm\" (UID: \"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5\") " pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:22.731024 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.730931    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e9c9fa47-a912-49cc-8c2f-bf6a60df68a5-proc\") pod \"perf-node-gather-daemonset-zmdkm\" (UID: \"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5\") " pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:22.731024 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.730958    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9c9fa47-a912-49cc-8c2f-bf6a60df68a5-sys\") pod \"perf-node-gather-daemonset-zmdkm\" (UID: \"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5\") " pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:22.832028 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.831997    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e9c9fa47-a912-49cc-8c2f-bf6a60df68a5-proc\") pod \"perf-node-gather-daemonset-zmdkm\" (UID: \"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5\") " pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:22.832028 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.832027    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9c9fa47-a912-49cc-8c2f-bf6a60df68a5-sys\") pod \"perf-node-gather-daemonset-zmdkm\" (UID: \"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5\") " pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:22.832219 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.832075    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e9c9fa47-a912-49cc-8c2f-bf6a60df68a5-podres\") pod \"perf-node-gather-daemonset-zmdkm\" (UID: \"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5\") " pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:22.832219 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.832112    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9c9fa47-a912-49cc-8c2f-bf6a60df68a5-lib-modules\") pod \"perf-node-gather-daemonset-zmdkm\" (UID: \"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5\") " pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:22.832219 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.832117    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e9c9fa47-a912-49cc-8c2f-bf6a60df68a5-proc\") pod \"perf-node-gather-daemonset-zmdkm\" (UID: \"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5\") " pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:22.832219 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.832132    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fcmj\" (UniqueName: \"kubernetes.io/projected/e9c9fa47-a912-49cc-8c2f-bf6a60df68a5-kube-api-access-8fcmj\") pod \"perf-node-gather-daemonset-zmdkm\" (UID: \"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5\") " pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:22.832219 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.832194    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9c9fa47-a912-49cc-8c2f-bf6a60df68a5-sys\") pod \"perf-node-gather-daemonset-zmdkm\" (UID: \"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5\") " pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:22.832387 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.832249    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e9c9fa47-a912-49cc-8c2f-bf6a60df68a5-podres\") pod \"perf-node-gather-daemonset-zmdkm\" (UID: \"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5\") " pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:22.832387 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.832258    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9c9fa47-a912-49cc-8c2f-bf6a60df68a5-lib-modules\") pod \"perf-node-gather-daemonset-zmdkm\" (UID: \"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5\") " pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:22.847530 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.847503    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fcmj\" (UniqueName: \"kubernetes.io/projected/e9c9fa47-a912-49cc-8c2f-bf6a60df68a5-kube-api-access-8fcmj\") pod \"perf-node-gather-daemonset-zmdkm\" (UID: \"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5\") " pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:22.941905 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:22.941821    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:23.079469 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:23.079438    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"]
Jul 23 14:29:23.082634 ip-10-0-138-125 kubenswrapper[2568]: W0723 14:29:23.082608    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode9c9fa47_a912_49cc_8c2f_bf6a60df68a5.slice/crio-470f32722e973c953d888e5370bf56cbd814d5729f52cd9d429b4305b0f3f0b3 WatchSource:0}: Error finding container 470f32722e973c953d888e5370bf56cbd814d5729f52cd9d429b4305b0f3f0b3: Status 404 returned error can't find the container with id 470f32722e973c953d888e5370bf56cbd814d5729f52cd9d429b4305b0f3f0b3
Jul 23 14:29:24.038523 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:24.038486    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm" event={"ID":"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5","Type":"ContainerStarted","Data":"96208f7f0b1b2fffe5815c845b6b691db302efa14ef1811443fd4ad0ed4e091d"}
Jul 23 14:29:24.038745 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:24.038542    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm" event={"ID":"e9c9fa47-a912-49cc-8c2f-bf6a60df68a5","Type":"ContainerStarted","Data":"470f32722e973c953d888e5370bf56cbd814d5729f52cd9d429b4305b0f3f0b3"}
Jul 23 14:29:24.038745 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:24.038568    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:24.060649 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:24.060593    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm" podStartSLOduration=2.060572207 podStartE2EDuration="2.060572207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 14:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 14:29:24.059952339 +0000 UTC m=+2912.521043383" watchObservedRunningTime="2026-07-23 14:29:24.060572207 +0000 UTC m=+2912.521663250"
Jul 23 14:29:24.778200 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:24.778171    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-54556fffb6-7p2wc_fc078541-1d9c-4eb6-963f-cf4102ade172/console/0.log"
Jul 23 14:29:26.591111 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:26.591086    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pfcfh_fb5287ae-1a85-49a7-8bff-7a6601b53bc2/dns/0.log"
Jul 23 14:29:26.612521 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:26.612494    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pfcfh_fb5287ae-1a85-49a7-8bff-7a6601b53bc2/kube-rbac-proxy/0.log"
Jul 23 14:29:26.796425 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:26.796395    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9tk9p_12dc50cb-2f68-4be0-b1a3-60e6dbb1cae5/dns-node-resolver/0.log"
Jul 23 14:29:27.506056 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:27.506027    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ddfgb_0489f3dd-bc92-4289-a74d-ceb26a4ee20e/node-ca/0.log"
Jul 23 14:29:28.817634 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:28.817607    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d57cbb598-cfjg4_ca94efb8-d44b-43d6-8d89-ef09eb258bf8/kube-auth-proxy/0.log"
Jul 23 14:29:29.735571 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:29.735522    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bgdb2_0f0aec49-2a44-4652-8c41-42471186cfc7/serve-healthcheck-canary/0.log"
Jul 23 14:29:30.053032 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:30.053002    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jrxnq/perf-node-gather-daemonset-zmdkm"
Jul 23 14:29:30.410357 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:30.410269    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z9dtb_2610fed0-d41b-49ba-aa2d-b5238ce82ab7/kube-rbac-proxy/0.log"
Jul 23 14:29:30.444250 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:30.444222    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z9dtb_2610fed0-d41b-49ba-aa2d-b5238ce82ab7/exporter/0.log"
Jul 23 14:29:30.477534 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:30.477512    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z9dtb_2610fed0-d41b-49ba-aa2d-b5238ce82ab7/extractor/0.log"
Jul 23 14:29:33.368595 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:33.368508    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d57cbb598-cfjg4_ca94efb8-d44b-43d6-8d89-ef09eb258bf8/kube-auth-proxy/0.log"
Jul 23 14:29:33.794517 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:33.794479    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c8b88fb69-4snf8_78a34ac0-ed10-48a3-ab9e-deb9e54c8989/manager/0.log"
Jul 23 14:29:33.873741 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:33.873714    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c8b88fb69-4snf8_78a34ac0-ed10-48a3-ab9e-deb9e54c8989/copy-manifests/0.log"
Jul 23 14:29:35.205533 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:35.205493    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d57cbb598-cfjg4_ca94efb8-d44b-43d6-8d89-ef09eb258bf8/kube-auth-proxy/0.log"
Jul 23 14:29:35.323139 ip-10-0-138-125 kubenswrapper[2568]: I0723 14:29:35.323106    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6d8d9c69b6-j5swr_b88a3fe2-f5f7-4da0-bb14-6c11561cf340/manager/0.log"