Jul 16 04:49:56.452894 ip-10-0-138-199 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 04:49:56.909020 ip-10-0-138-199 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 04:49:56.909020 ip-10-0-138-199 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 04:49:56.909020 ip-10-0-138-199 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 04:49:56.909020 ip-10-0-138-199 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 04:49:56.909020 ip-10-0-138-199 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 04:49:56.910114 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.910027    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 04:49:56.914955 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.914936    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 04:49:56.914955 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.914956    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.914960    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.914964    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.914967    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.914972    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.914976    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.914979    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.914982    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.914985    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.914988    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.914990    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.914993    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.914995    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.914999    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915001    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915004    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915006    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915009    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915013    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 04:49:56.915026 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915016    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915021    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915024    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915027    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915031    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915034    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915037    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915040    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915043    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915045    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915048    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915050    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915053    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915055    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915057    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915060    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915062    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915064    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915067    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 04:49:56.915554 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915070    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915072    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915075    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915077    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915080    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915082    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915085    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915088    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915092    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915094    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915097    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915099    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915102    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915104    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915107    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915110    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915112    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915115    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915117    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 04:49:56.916038 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915120    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915122    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915125    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915127    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915129    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915132    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915135    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915137    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915140    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915142    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915145    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915148    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915150    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915153    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915156    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915158    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915160    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915163    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915166    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915169    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 04:49:56.916496 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915172    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915174    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915177    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915181    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915183    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915186    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915188    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915191    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915193    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915195    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915198    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915201    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915203    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915206    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915208    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915211    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915213    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915216    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915218    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915221    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 04:49:56.916988 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915223    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915963    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915970    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915973    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915976    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915979    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915982    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915984    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915987    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915990    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915992    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915996    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.915998    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916001    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916003    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916007    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916009    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916012    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916014    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916019    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 04:49:56.917465 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916023    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916025    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916029    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916032    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916035    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916038    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916041    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916044    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916046    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916049    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916052    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916055    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916057    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916060    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916063    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916066    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916069    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916072    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916074    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 04:49:56.917969 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916079    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916081    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916084    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916086    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916091    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916096    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916100    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916103    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916106    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916109    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916112    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916115    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916117    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916120    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916122    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916125    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916127    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916130    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916132    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 04:49:56.918438 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916136    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916139    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916142    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916144    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916146    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916149    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916152    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916154    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916156    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916159    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916161    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916164    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916166    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916168    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916172    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916174    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916177    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916180    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916184    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916186    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 04:49:56.918921 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916189    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916191    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916194    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916197    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916199    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916201    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916204    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916206    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916209    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916211    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916213    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916216    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916218    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916220    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916223    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916225    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916228    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916230    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916232    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916235    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 04:49:56.919407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916237    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916240    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916641    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916648    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916652    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916656    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916660    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916663    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916666    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916670    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916673    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916676    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916678    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916681    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916683    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916686    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916688    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916691    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916693    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916695    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 04:49:56.919909 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916698    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916701    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916703    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916705    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916708    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916710    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916713    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916715    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916718    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916720    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916722    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916725    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916727    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916729    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916732    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916734    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916737    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916740    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916743    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 04:49:56.920395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916746    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916749    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916752    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916756    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916759    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916761    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916764    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916767    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916769    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916772    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916774    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916777    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916779    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916782    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916784    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916786    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916789    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916791    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916794    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 04:49:56.920891 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916796    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916798    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916801    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916803    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916805    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916808    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916810    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916813    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916815    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916817    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916820    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916824    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916828    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916830    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916833    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916837    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916839    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916843    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916846    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916848    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 04:49:56.921339 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916851    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916853    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916856    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916858    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916861    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916863    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916866    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916868    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916870    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916873    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916875    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916878    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916880    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916883    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916885    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916889    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916891    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916894    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916896    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 04:49:56.921832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916899    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916902    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916904    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.916906    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.916980    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.916987    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.916995    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917002    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917011    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917016    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917021    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917026    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917030    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917033    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917036    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917039    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917042    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917045    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917048    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917051    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917054    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917056    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917062    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917064    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 04:49:56.922327 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917067    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917070    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917073    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917077    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917080    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917083    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917086    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917089    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917092    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917096    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917098    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917101    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917107    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917110    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917113    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917116    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917119    2565 flags.go:64] FLAG: --enable-server="true"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917122    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917126    2565 flags.go:64] FLAG: --event-burst="100"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917130    2565 flags.go:64] FLAG: --event-qps="50"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917132    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917135    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917138    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917142    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 04:49:56.922924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917144    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917147    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917150    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917152    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917155    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917158    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917161    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917163    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917166    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917169    2565 flags.go:64] FLAG: --feature-gates=""
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917177    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917180    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917183    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917187    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917190    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917193    2565 flags.go:64] FLAG: --help="false"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917196    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-138-199.ec2.internal"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917199    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917202    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917204    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917209    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917212    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917215    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 04:49:56.923540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917218    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917221    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917224    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917227    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917230    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917233    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917236    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917239    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917242    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917245    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917247    2565 flags.go:64] FLAG: --lock-file=""
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917250    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917253    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917256    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917261    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917264    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917267    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917269    2565 flags.go:64] FLAG: --logging-format="text"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917272    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917275    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917278    2565 flags.go:64] FLAG: --manifest-url=""
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917281    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917285    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917288    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917294    2565 flags.go:64] FLAG: --max-pods="110"
Jul 16 04:49:56.924130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917297    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917299    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917302    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917305    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917309    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917312    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917314    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917322    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917325    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917346    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917349    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917353    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917359    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917363    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917366    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917369    2565 flags.go:64] FLAG: --port="10250"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917372    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917375    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cc02a8081d9487f1"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917378    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917381    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917383    2565 flags.go:64] FLAG: --register-node="true"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917386    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917390    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917392    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 16 04:49:56.924759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917395    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917398    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917402    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917405    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917408    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917411    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917414    2565 flags.go:64] FLAG: --runonce="false"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917418    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917421    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917423    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917427    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917429    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917432    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917437    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917440    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917443    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917445    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917448    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917451    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917454    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917457    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917460    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917465    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917468    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917471    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 04:49:56.925349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917476    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917479    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917482    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917485    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917488    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917491    2565 flags.go:64] FLAG: --v="2"
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917495    2565 flags.go:64] FLAG: --version="false"
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917499    2565 flags.go:64] FLAG: --vmodule=""
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917503    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917506    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917646    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917652    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917655    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917659    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917664    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917666    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917669    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917671    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917674    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917677    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917681    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917683    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 04:49:56.925964 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917686    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917688    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917692    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917694    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917697    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917700    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917702    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917705    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917708    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917710    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917714    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917718    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917721    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917724    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917726    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917729    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917732    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917735    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917738    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 04:49:56.926517 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917740    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917743    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917745    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917748    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917750    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917754    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917757    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917759    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917761    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917764    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917766    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917771    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917774    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917778    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917780    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917783    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917785    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917788    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917791    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 04:49:56.927097 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917794    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917797    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917799    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917802    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917804    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917807    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917809    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917812    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917814    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917817    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917819    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917821    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917824    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917826    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917830    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917833    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917836    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917838    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917842    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917845    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 04:49:56.927625 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917848    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917850    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917853    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917855    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917859    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917862    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917864    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917867    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917869    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917871    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917874    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917876    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917879    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917882    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917884    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917887    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917889    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917892    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917894    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917897    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 04:49:56.928114 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917899    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 04:49:56.928621 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917902    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 04:49:56.928621 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917904    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 04:49:56.928621 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917907    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 04:49:56.928621 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917909    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 04:49:56.928621 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917912    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 04:49:56.928621 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917915    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 04:49:56.928621 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917917    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 04:49:56.928621 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.917920    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 04:49:56.928621 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.917929    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 04:49:56.928621 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.926165    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 04:49:56.928621 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.926189    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 04:49:56.928621 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.927048    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 04:49:56.928621 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.927056    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 04:49:56.928621 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927131    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927138    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927142    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927145    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927148    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927151    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927154    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927157    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927160    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927162    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927165    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927168    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927171    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927173    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927175    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927179    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927181    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927185    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927189    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927191    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 04:49:56.928979 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927194    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927196    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927199    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927201    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927204    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927207    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927209    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927212    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927214    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927225    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927228    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927230    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927233    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927235    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927238    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927241    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927243    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927246    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927248    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 04:49:56.929464 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927251    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927254    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927256    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927259    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927261    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927264    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927266    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927269    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927271    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927273    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927276    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927278    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927281    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927285    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927288    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927291    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927294    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927297    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927301    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 04:49:56.929945 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927304    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927307    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927310    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927314    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927323    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927327    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927330    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927332    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927335    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927337    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927345    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927348    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927350    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927353    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927356    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927358    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927361    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927363    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927365    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927368    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 04:49:56.930412 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927371    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927373    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927376    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927379    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927381    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927383    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927386    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927388    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927391    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927393    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927396    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927399    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927401    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927404    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927406    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927410    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927412    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927421    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927424    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927426    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 04:49:56.930912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927428    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 04:49:56.931397 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.927434    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 04:49:56.931397 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927616    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 04:49:56.931397 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927624    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 04:49:56.931397 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927627    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 04:49:56.931397 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927630    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 04:49:56.931397 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927633    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 04:49:56.931397 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927636    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 04:49:56.931397 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927638    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 04:49:56.931397 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927641    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 04:49:56.931397 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927644    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 04:49:56.931397 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927646    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 04:49:56.931397 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927650    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 04:49:56.931397 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927652    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 04:49:56.931397 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927655    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 04:49:56.931397 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927657    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927660    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927662    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927665    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927667    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927670    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927672    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927675    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927677    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927681    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927685    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927687    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927689    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927692    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927695    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927707    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927710    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927712    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927715    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 04:49:56.931782 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927717    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927720    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927722    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927725    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927727    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927730    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927732    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927735    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927737    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927741    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927745    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927748    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927751    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927754    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927756    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927759    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927762    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927765    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927768    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927770    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 04:49:56.932238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927773    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927775    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927778    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927780    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927783    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927785    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927788    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927790    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927792    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927802    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927806    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927808    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927811    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927814    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927816    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927819    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927821    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927823    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927826    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 04:49:56.932832 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927829    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927831    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927834    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927836    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927838    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927841    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927843    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927846    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927848    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927850    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927853    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927855    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927858    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927860    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927862    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927865    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927867    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927870    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927873    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927875    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 04:49:56.933315 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927877    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 04:49:56.933813 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927880    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 04:49:56.933813 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927882    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 04:49:56.933813 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927890    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 04:49:56.933813 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927893    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 04:49:56.933813 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927895    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 04:49:56.933813 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927898    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 04:49:56.933813 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:56.927900    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 04:49:56.933813 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.927906    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 04:49:56.933813 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.928052    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 04:49:56.933813 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.930763    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 04:49:56.933813 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.932565    2565 server.go:1019] "Starting client certificate rotation"
Jul 16 04:49:56.933813 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.932756    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 04:49:56.933813 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.932794    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 04:49:56.938402 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.938377    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 04:49:56.939773 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.939748    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 04:49:56.954661 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.954630    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 16 04:49:56.957239 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.957225    2565 log.go:25] "Validated CRI v1 image API"
Jul 16 04:49:56.958106 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.958085    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 04:49:56.960575 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.960554    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 04:49:56.960979 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.960961    2565 fs.go:135] Filesystem UUIDs: map[1a018152-4106-4abe-aa8c-8c3b9a650b5e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b7331e85-8f76-455b-adc4-69d20519f582:/dev/nvme0n1p4]
Jul 16 04:49:56.961018 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.960980    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 04:49:56.967965 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.967843    2565 manager.go:216] Machine: {Timestamp:2026-07-16 04:49:56.96633982 +0000 UTC m=+0.382294886 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3113918 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f184fc215a35d499f1d0df06fc27d SystemUUID:ec2f184f-c215-a35d-499f-1d0df06fc27d BootID:8d8dddbe-63d9-46f8-a8d7-b70c162d2479 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:87:3a:9d:ca:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:87:3a:9d:ca:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:b2:97:3b:1d:c4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 04:49:56.967965 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.967961    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 04:49:56.968085 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.968060    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 04:49:56.971884 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.971850    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 04:49:56.972035 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.971887    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-199.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 04:49:56.972756 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.972744    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 04:49:56.972789 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.972758    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 04:49:56.972789 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.972766    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 04:49:56.972839 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.972793    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 04:49:56.974707 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.974697    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 04:49:56.979483 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.979468    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 04:49:56.979655 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.979645    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 04:49:56.979696 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.979680    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 04:49:56.979753 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.979700    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 04:49:56.979753 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.979715    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 04:49:56.979753 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.979728    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 16 04:49:56.979753 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.979738    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 04:49:56.980197 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.980175    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 04:49:56.980248 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.980202    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 04:49:56.982721 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.982702    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 04:49:56.985051 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.985037    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 04:49:56.985109 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.985069    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 04:49:56.985283 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.985252    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 04:49:56.985283 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.985270    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 04:49:56.985283 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.985277    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 04:49:56.985450 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.985292    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 04:49:56.985450 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.985304    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 04:49:56.985450 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.985310    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 04:49:56.986149 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.986127    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 04:49:56.986221 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.986153    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 04:49:56.986221 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.986167    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 04:49:56.986221 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.986178    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 04:49:56.986221 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.986192    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 04:49:56.986221 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.986205    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 04:49:56.987060 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.987040    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 04:49:56.987133 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.987063    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 04:49:56.990733 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.990712    2565 server.go:1268] "Started kubelet"
Jul 16 04:49:56.990854 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.990825    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 04:49:56.990943 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.990893    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 04:49:56.990991 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.990965    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 04:49:56.991450 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.991427    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-199.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 04:49:56.991746 ip-10-0-138-199 systemd[1]: Started Kubernetes Kubelet.
Jul 16 04:49:56.992393 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:56.992177    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 04:49:56.992393 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:56.992179    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-199.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 04:49:56.992973 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.992859    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 04:49:56.993038 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:56.993026    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 04:49:56.997742 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:56.997722    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 04:49:56.998155 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:56.996943    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-199.ec2.internal.18c2abe3bbe91f9a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-199.ec2.internal,UID:ip-10-0-138-199.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-199.ec2.internal,},FirstTimestamp:2026-07-16 04:49:56.990680986 +0000 UTC m=+0.406636050,LastTimestamp:2026-07-16 04:49:56.990680986 +0000 UTC m=+0.406636050,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-199.ec2.internal,}"
Jul 16 04:49:57.000948 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.000930    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 04:49:57.001031 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.000937    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 04:49:57.001091 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.001049    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-199.ec2.internal\" not found"
Jul 16 04:49:57.001764 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.001749    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 04:49:57.001764 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.001765    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 04:49:57.001979 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.001795    2565 factory.go:55] Registering systemd factory
Jul 16 04:49:57.001979 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.001809    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 16 04:49:57.001979 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.001918    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 04:49:57.001979 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.001927    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 04:49:57.002482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.002459    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 04:49:57.004002 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.003373    2565 factory.go:153] Registering CRI-O factory
Jul 16 04:49:57.004002 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.003390    2565 factory.go:223] Registration of the crio container factory successfully
Jul 16 04:49:57.004002 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.003458    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 04:49:57.004002 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.003481    2565 factory.go:103] Registering Raw factory
Jul 16 04:49:57.004002 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.003496    2565 manager.go:1196] Started watching for new ooms in manager
Jul 16 04:49:57.004002 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.003885    2565 manager.go:349] Starting recovery of all containers
Jul 16 04:49:57.005526 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.005498    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cff27"
Jul 16 04:49:57.010497 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.010321    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 04:49:57.010664 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.010640    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-199.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 04:49:57.011683 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.011667    2565 manager.go:354] Recovery completed
Jul 16 04:49:57.013151 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.013127    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cff27"
Jul 16 04:49:57.018441 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.018421    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 04:49:57.020646 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.020624    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 04:49:57.020729 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.020657    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 04:49:57.020729 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.020669    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasSufficientPID"
Jul 16 04:49:57.021263 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.021250    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 04:49:57.021263 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.021262    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 04:49:57.021348 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.021280    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 04:49:57.021348 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.021261    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 04:49:57.021973 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.021960    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 04:49:57.022023 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.021987    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 04:49:57.022023 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.022001    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasSufficientPID"
Jul 16 04:49:57.022584 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.022515    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-199.ec2.internal.18c2abe3bdb254b3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-199.ec2.internal,UID:ip-10-0-138-199.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-199.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-199.ec2.internal,},FirstTimestamp:2026-07-16 04:49:57.020644531 +0000 UTC m=+0.436599597,LastTimestamp:2026-07-16 04:49:57.020644531 +0000 UTC m=+0.436599597,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-199.ec2.internal,}"
Jul 16 04:49:57.023255 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.023242    2565 policy_none.go:49] "None policy: Start"
Jul 16 04:49:57.023295 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.023260    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 04:49:57.023295 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.023271    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 04:49:57.025302 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.025292    2565 policy_none.go:47] "Start"
Jul 16 04:49:57.063529 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.063510    2565 manager.go:337] "Starting Device Plugin manager"
Jul 16 04:49:57.087796 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.063546    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 04:49:57.087796 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.063556    2565 server.go:87] "Starting device plugin registration server"
Jul 16 04:49:57.087796 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.063827    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 04:49:57.087796 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.063837    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 04:49:57.087796 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.063957    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 04:49:57.087796 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.064059    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 04:49:57.087796 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.064070    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 04:49:57.087796 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.064523    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 04:49:57.087796 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.064579    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-199.ec2.internal\" not found"
Jul 16 04:49:57.133710 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.133660    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 04:49:57.135084 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.135064    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 04:49:57.135174 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.135114    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 04:49:57.135174 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.135145    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 04:49:57.135252 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.135191    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 04:49:57.138119 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.138096    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 04:49:57.164793 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.164708    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 04:49:57.165972 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.165935    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 04:49:57.165972 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.165976    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 04:49:57.166132 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.165990    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasSufficientPID"
Jul 16 04:49:57.166132 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.166014    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.177306 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.177276    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.177306 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.177307    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-199.ec2.internal\": node \"ip-10-0-138-199.ec2.internal\" not found"
Jul 16 04:49:57.195634 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.195583    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-199.ec2.internal\" not found"
Jul 16 04:49:57.236171 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.236136    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-199.ec2.internal"]
Jul 16 04:49:57.236264 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.236219    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 04:49:57.237928 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.237902    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 04:49:57.238057 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.237939    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 04:49:57.238057 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.237949    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasSufficientPID"
Jul 16 04:49:57.240164 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.240148    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 04:49:57.240285 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.240269    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.240343 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.240308    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 04:49:57.241050 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.241034    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 04:49:57.241139 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.241062    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 04:49:57.241139 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.241072    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 04:49:57.241139 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.241095    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 04:49:57.241139 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.241108    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasSufficientPID"
Jul 16 04:49:57.241139 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.241074    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasSufficientPID"
Jul 16 04:49:57.243363 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.243344    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.243452 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.243377    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 04:49:57.244168 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.244148    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 04:49:57.244168 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.244171    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 04:49:57.244317 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.244180    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeHasSufficientPID"
Jul 16 04:49:57.273557 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.273529    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-199.ec2.internal\" not found" node="ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.278378 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.278358    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-199.ec2.internal\" not found" node="ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.296524 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.296497    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-199.ec2.internal\" not found"
Jul 16 04:49:57.302875 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.302841    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4fb9095ab1b7993f0bf6be0862d6ba52-config\") pod \"kube-apiserver-proxy-ip-10-0-138-199.ec2.internal\" (UID: \"4fb9095ab1b7993f0bf6be0862d6ba52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.302950 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.302883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/85a4e4c8efa90150c8951f2c413ffb17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal\" (UID: \"85a4e4c8efa90150c8951f2c413ffb17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.302950 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.302902    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85a4e4c8efa90150c8951f2c413ffb17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal\" (UID: \"85a4e4c8efa90150c8951f2c413ffb17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.396671 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.396639    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-199.ec2.internal\" not found"
Jul 16 04:49:57.404031 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.404002    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/85a4e4c8efa90150c8951f2c413ffb17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal\" (UID: \"85a4e4c8efa90150c8951f2c413ffb17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.404099 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.404039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85a4e4c8efa90150c8951f2c413ffb17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal\" (UID: \"85a4e4c8efa90150c8951f2c413ffb17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.404099 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.404056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4fb9095ab1b7993f0bf6be0862d6ba52-config\") pod \"kube-apiserver-proxy-ip-10-0-138-199.ec2.internal\" (UID: \"4fb9095ab1b7993f0bf6be0862d6ba52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.404172 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.404101    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4fb9095ab1b7993f0bf6be0862d6ba52-config\") pod \"kube-apiserver-proxy-ip-10-0-138-199.ec2.internal\" (UID: \"4fb9095ab1b7993f0bf6be0862d6ba52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.404172 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.404098    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/85a4e4c8efa90150c8951f2c413ffb17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal\" (UID: \"85a4e4c8efa90150c8951f2c413ffb17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.404172 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.404102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85a4e4c8efa90150c8951f2c413ffb17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal\" (UID: \"85a4e4c8efa90150c8951f2c413ffb17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.497524 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.497451    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-199.ec2.internal\" not found"
Jul 16 04:49:57.575012 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.574976    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.581214 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.581197    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-199.ec2.internal"
Jul 16 04:49:57.597997 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.597968    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-199.ec2.internal\" not found"
Jul 16 04:49:57.699107 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.699063    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-199.ec2.internal\" not found"
Jul 16 04:49:57.799704 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.799634    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-199.ec2.internal\" not found"
Jul 16 04:49:57.900268 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:57.900232    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-199.ec2.internal\" not found"
Jul 16 04:49:57.931843 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.931811    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 04:49:57.932438 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:57.931985    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 04:49:58.000378 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:58.000351    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-199.ec2.internal\" not found"
Jul 16 04:49:58.018427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.018389    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 04:44:57 +0000 UTC" deadline="2028-01-23 03:06:30.263427556 +0000 UTC"
Jul 16 04:49:58.018427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.018420    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13342h16m32.245010352s"
Jul 16 04:49:58.100786 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:58.100747    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-199.ec2.internal\" not found"
Jul 16 04:49:58.120688 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.120655    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 04:49:58.166012 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.165989    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 04:49:58.201316 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.201290    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal"
Jul 16 04:49:58.207823 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:58.207773    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb9095ab1b7993f0bf6be0862d6ba52.slice/crio-d9f2a486f491b00c516c4b9a3fee8595d929802c761338923f7a1a8cf087eecf WatchSource:0}: Error finding container d9f2a486f491b00c516c4b9a3fee8595d929802c761338923f7a1a8cf087eecf: Status 404 returned error can't find the container with id d9f2a486f491b00c516c4b9a3fee8595d929802c761338923f7a1a8cf087eecf
Jul 16 04:49:58.208193 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:58.208167    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a4e4c8efa90150c8951f2c413ffb17.slice/crio-b7f7b71a73a5c1d1998d538e8a76983bfe24a80598d10a86f9bb07c160ffded6 WatchSource:0}: Error finding container b7f7b71a73a5c1d1998d538e8a76983bfe24a80598d10a86f9bb07c160ffded6: Status 404 returned error can't find the container with id b7f7b71a73a5c1d1998d538e8a76983bfe24a80598d10a86f9bb07c160ffded6
Jul 16 04:49:58.212111 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.212084    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 04:49:58.214888 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.214870    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 04:49:58.216437 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.216421    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-199.ec2.internal"
Jul 16 04:49:58.223680 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.223655    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 04:49:58.331440 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.331351    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 04:49:58.980612 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.980549    2565 apiserver.go:52] "Watching apiserver"
Jul 16 04:49:58.989444 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.989408    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 04:49:58.989837 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.989805    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-9fsdp","openshift-multus/network-metrics-daemon-ldxlj","openshift-network-diagnostics/network-check-target-pz5t7","openshift-network-operator/iptables-alerter-dk659","openshift-ovn-kubernetes/ovnkube-node-5nsc5","kube-system/global-pull-secret-syncer-dmm65","kube-system/konnectivity-agent-5rnd9","kube-system/kube-apiserver-proxy-ip-10-0-138-199.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c","openshift-cluster-node-tuning-operator/tuned-w5m9p","openshift-dns/node-resolver-cbrhk","openshift-multus/multus-t8rkk","openshift-image-registry/node-ca-sslfh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal"]
Jul 16 04:49:58.992818 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.992788    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5rnd9"
Jul 16 04:49:58.995224 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.995195    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sslfh"
Jul 16 04:49:58.995409 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.995397    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rtl9h\""
Jul 16 04:49:58.995704 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.995682    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 04:49:58.995821 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.995801    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 04:49:58.999536 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.999513    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 04:49:58.999948 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.999810    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5wlqb\""
Jul 16 04:49:58.999948 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.999934    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 04:49:59.000110 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:58.999975    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 04:49:59.000642 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.000163    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:49:59.000642 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.000239    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:49:59.000642 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.000453    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dk659"
Jul 16 04:49:59.001752 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.001643    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 04:49:59.002977 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.002941    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 04:49:59.003216 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.003197    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.003265 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.003246    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 04:49:59.003537 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.003517    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mknws\""
Jul 16 04:49:59.003766 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.003744    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 04:49:59.005946 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.005538    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:49:59.005946 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.005610    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:49:59.006113 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.005997    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 04:49:59.006181 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.006164    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 04:49:59.006838 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.006711    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hmdbw\""
Jul 16 04:49:59.006838 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.006719    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 04:49:59.007785 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.007761    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 04:49:59.007901 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.007839    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 04:49:59.007901 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.007875    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 04:49:59.008493 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.008469    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.011253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.011232    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.012083 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.011817    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 04:49:59.012083 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.011825    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 04:49:59.012567 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.012450    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-875rr\""
Jul 16 04:49:59.012567 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.012466    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.013969    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2066081-de65-45f1-a46c-0f456249f49e-serviceca\") pod \"node-ca-sslfh\" (UID: \"a2066081-de65-45f1-a46c-0f456249f49e\") " pod="openshift-image-registry/node-ca-sslfh"
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014006    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-kubelet\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014036    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-run-openvswitch\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014055    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014064    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-run-ovn-kubernetes\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014091    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-run-systemd\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014116    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-cni-bin\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014139    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-systemd-units\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014182    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgp8x\" (UniqueName: \"kubernetes.io/projected/dedd46bb-de3b-4d76-9276-9bc6ac152348-kube-api-access-dgp8x\") pod \"iptables-alerter-dk659\" (UID: \"dedd46bb-de3b-4d76-9276-9bc6ac152348\") " pod="openshift-network-operator/iptables-alerter-dk659"
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014210    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-var-lib-openvswitch\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014236    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-log-socket\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014261    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8nhc\" (UniqueName: \"kubernetes.io/projected/72b01c84-01a4-4022-b66f-95876da5da22-kube-api-access-n8nhc\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014287    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dedd46bb-de3b-4d76-9276-9bc6ac152348-host-slash\") pod \"iptables-alerter-dk659\" (UID: \"dedd46bb-de3b-4d76-9276-9bc6ac152348\") " pod="openshift-network-operator/iptables-alerter-dk659"
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-run-netns\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014340    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-etc-openvswitch\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014357    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 04:49:59.015304 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014380    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72b01c84-01a4-4022-b66f-95876da5da22-ovnkube-script-lib\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014407    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a3625d5-1418-40d6-b16b-1cb8a069163a-agent-certs\") pod \"konnectivity-agent-5rnd9\" (UID: \"5a3625d5-1418-40d6-b16b-1cb8a069163a\") " pod="kube-system/konnectivity-agent-5rnd9"
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014429    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2066081-de65-45f1-a46c-0f456249f49e-host\") pod \"node-ca-sslfh\" (UID: \"a2066081-de65-45f1-a46c-0f456249f49e\") " pod="openshift-image-registry/node-ca-sslfh"
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014458    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rdr9\" (UniqueName: \"kubernetes.io/projected/a2066081-de65-45f1-a46c-0f456249f49e-kube-api-access-7rdr9\") pod \"node-ca-sslfh\" (UID: \"a2066081-de65-45f1-a46c-0f456249f49e\") " pod="openshift-image-registry/node-ca-sslfh"
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014491    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-node-log\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72b01c84-01a4-4022-b66f-95876da5da22-ovn-node-metrics-cert\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014546    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql6fr\" (UniqueName: \"kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr\") pod \"network-check-target-pz5t7\" (UID: \"b74a6491-0f27-40ee-8f69-b0480a1d1889\") " pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-run-ovn\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014630    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72b01c84-01a4-4022-b66f-95876da5da22-env-overrides\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014673    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a3625d5-1418-40d6-b16b-1cb8a069163a-konnectivity-ca\") pod \"konnectivity-agent-5rnd9\" (UID: \"5a3625d5-1418-40d6-b16b-1cb8a069163a\") " pod="kube-system/konnectivity-agent-5rnd9"
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014700    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dedd46bb-de3b-4d76-9276-9bc6ac152348-iptables-alerter-script\") pod \"iptables-alerter-dk659\" (UID: \"dedd46bb-de3b-4d76-9276-9bc6ac152348\") " pod="openshift-network-operator/iptables-alerter-dk659"
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014750    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-slash\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014776    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-cni-netd\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72b01c84-01a4-4022-b66f-95876da5da22-ovnkube-config\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014817    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dbp4v\""
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014905    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.015067    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014819    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 04:49:59.016091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.014908    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:49:59.017031 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.015239    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:49:59.018009 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.017902    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 04:49:59.018257 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.018182    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.026709 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.026625    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cbrhk"
Jul 16 04:49:59.029480 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.029389    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 04:49:59.029675 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.029649    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 04:49:59.029793 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.029724    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ghc4r\""
Jul 16 04:49:59.030701 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.030054    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5fsvh\""
Jul 16 04:49:59.030701 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.030301    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.030701 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.030400    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 04:49:59.031074 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.031048    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 04:49:59.032979 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.032817    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 04:49:59.032979 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.032861    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wgql7\""
Jul 16 04:49:59.043374 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.043345    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x2zd2"
Jul 16 04:49:59.051296 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.051260    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x2zd2"
Jul 16 04:49:59.103908 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.103873    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.115945    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-log-socket\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.115994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8nhc\" (UniqueName: \"kubernetes.io/projected/72b01c84-01a4-4022-b66f-95876da5da22-kube-api-access-n8nhc\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116028    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-systemd\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116044    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-log-socket\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116051    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-var-lib-kubelet\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116102    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-etc-kubernetes\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116131    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kgwf\" (UniqueName: \"kubernetes.io/projected/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-kube-api-access-8kgwf\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116150    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74612e83-5ffe-4c5d-830b-8e3f83fd8579-dbus\") pod \"global-pull-secret-syncer-dmm65\" (UID: \"74612e83-5ffe-4c5d-830b-8e3f83fd8579\") " pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116172    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec34c160-0104-4e3e-9e81-bd84e8266708-system-cni-dir\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116208    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhsn\" (UniqueName: \"kubernetes.io/projected/4d3ed278-258b-4590-846b-c11283cbbba3-kube-api-access-vdhsn\") pod \"node-resolver-cbrhk\" (UID: \"4d3ed278-258b-4590-846b-c11283cbbba3\") " pod="openshift-dns/node-resolver-cbrhk"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116236    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-sysctl-d\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116266    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-sys\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116293    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2066081-de65-45f1-a46c-0f456249f49e-host\") pod \"node-ca-sslfh\" (UID: \"a2066081-de65-45f1-a46c-0f456249f49e\") " pod="openshift-image-registry/node-ca-sslfh"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rdr9\" (UniqueName: \"kubernetes.io/projected/a2066081-de65-45f1-a46c-0f456249f49e-kube-api-access-7rdr9\") pod \"node-ca-sslfh\" (UID: \"a2066081-de65-45f1-a46c-0f456249f49e\") " pod="openshift-image-registry/node-ca-sslfh"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116336    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72b01c84-01a4-4022-b66f-95876da5da22-ovn-node-metrics-cert\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-device-dir\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.116451 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116405    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.117305 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116440    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec34c160-0104-4e3e-9e81-bd84e8266708-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.117305 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116497    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2066081-de65-45f1-a46c-0f456249f49e-host\") pod \"node-ca-sslfh\" (UID: \"a2066081-de65-45f1-a46c-0f456249f49e\") " pod="openshift-image-registry/node-ca-sslfh"
Jul 16 04:49:59.117305 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116641    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-multus-cni-dir\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.117305 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-os-release\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.117305 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116711    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6fr\" (UniqueName: \"kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr\") pod \"network-check-target-pz5t7\" (UID: \"b74a6491-0f27-40ee-8f69-b0480a1d1889\") " pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:49:59.117305 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.117305 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116773    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74612e83-5ffe-4c5d-830b-8e3f83fd8579-kubelet-config\") pod \"global-pull-secret-syncer-dmm65\" (UID: \"74612e83-5ffe-4c5d-830b-8e3f83fd8579\") " pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:49:59.117305 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x24p\" (UniqueName: \"kubernetes.io/projected/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-kube-api-access-7x24p\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.117305 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116856    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3ed278-258b-4590-846b-c11283cbbba3-tmp-dir\") pod \"node-resolver-cbrhk\" (UID: \"4d3ed278-258b-4590-846b-c11283cbbba3\") " pod="openshift-dns/node-resolver-cbrhk"
Jul 16 04:49:59.117305 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116879    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-sysctl-conf\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.117305 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116900    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 04:49:59.117305 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117148    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.117906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.116904    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf1738c9-f66f-4506-9b6e-422e9dd92071-tmp\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.117906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.117906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117376    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-multus-socket-dir-parent\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.117906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dedd46bb-de3b-4d76-9276-9bc6ac152348-iptables-alerter-script\") pod \"iptables-alerter-dk659\" (UID: \"dedd46bb-de3b-4d76-9276-9bc6ac152348\") " pod="openshift-network-operator/iptables-alerter-dk659"
Jul 16 04:49:59.117906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-slash\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.117906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117453    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qvt5\" (UniqueName: \"kubernetes.io/projected/ec34c160-0104-4e3e-9e81-bd84e8266708-kube-api-access-5qvt5\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.117906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-slash\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.117906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117549    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvjhg\" (UniqueName: \"kubernetes.io/projected/c028e133-a290-4758-998c-24e158389825-kube-api-access-kvjhg\") pod \"network-metrics-daemon-ldxlj\" (UID: \"c028e133-a290-4758-998c-24e158389825\") " pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:49:59.117906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-sysconfig\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.117906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117646    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-kubernetes\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.117906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117709    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-var-lib-cni-multus\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.117906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2066081-de65-45f1-a46c-0f456249f49e-serviceca\") pod \"node-ca-sslfh\" (UID: \"a2066081-de65-45f1-a46c-0f456249f49e\") " pod="openshift-image-registry/node-ca-sslfh"
Jul 16 04:49:59.117906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117765    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-sys-fs\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.117906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117808    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-var-lib-cni-bin\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.117906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117887    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-multus-daemon-config\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117917    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-run-systemd\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.117982    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret\") pod \"global-pull-secret-syncer-dmm65\" (UID: \"74612e83-5ffe-4c5d-830b-8e3f83fd8579\") " pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs\") pod \"network-metrics-daemon-ldxlj\" (UID: \"c028e133-a290-4758-998c-24e158389825\") " pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-run\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118075    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-host\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-cni-binary-copy\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118153    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-run-k8s-cni-cncf-io\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118176    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dedd46bb-de3b-4d76-9276-9bc6ac152348-iptables-alerter-script\") pod \"iptables-alerter-dk659\" (UID: \"dedd46bb-de3b-4d76-9276-9bc6ac152348\") " pod="openshift-network-operator/iptables-alerter-dk659"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-systemd-units\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118316    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-var-lib-openvswitch\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118345    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec34c160-0104-4e3e-9e81-bd84e8266708-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118387    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-lib-modules\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118417    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dedd46bb-de3b-4d76-9276-9bc6ac152348-host-slash\") pod \"iptables-alerter-dk659\" (UID: \"dedd46bb-de3b-4d76-9276-9bc6ac152348\") " pod="openshift-network-operator/iptables-alerter-dk659"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-run-netns\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118479    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-etc-openvswitch\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118503    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72b01c84-01a4-4022-b66f-95876da5da22-ovnkube-script-lib\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.118657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118526    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-registration-dir\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118581    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a3625d5-1418-40d6-b16b-1cb8a069163a-agent-certs\") pod \"konnectivity-agent-5rnd9\" (UID: \"5a3625d5-1418-40d6-b16b-1cb8a069163a\") " pod="kube-system/konnectivity-agent-5rnd9"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-node-log\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118671    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-tuned\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118714    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-run-ovn\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118743    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec34c160-0104-4e3e-9e81-bd84e8266708-cnibin\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118802    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec34c160-0104-4e3e-9e81-bd84e8266708-os-release\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.118836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec34c160-0104-4e3e-9e81-bd84e8266708-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119036    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-system-cni-dir\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119072    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72b01c84-01a4-4022-b66f-95876da5da22-env-overrides\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119117    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xx9z\" (UniqueName: \"kubernetes.io/projected/bf1738c9-f66f-4506-9b6e-422e9dd92071-kube-api-access-4xx9z\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119144    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-run-multus-certs\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a3625d5-1418-40d6-b16b-1cb8a069163a-konnectivity-ca\") pod \"konnectivity-agent-5rnd9\" (UID: \"5a3625d5-1418-40d6-b16b-1cb8a069163a\") " pod="kube-system/konnectivity-agent-5rnd9"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119211    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-cni-netd\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119237    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72b01c84-01a4-4022-b66f-95876da5da22-ovnkube-config\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119264    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec34c160-0104-4e3e-9e81-bd84e8266708-cni-binary-copy\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119304    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-run-netns\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.119482 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-hostroot\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.120062 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119373    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-kubelet\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.120062 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-run-openvswitch\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.120062 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119456    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-run-ovn-kubernetes\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.120062 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-socket-dir\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.120062 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d3ed278-258b-4590-846b-c11283cbbba3-hosts-file\") pod \"node-resolver-cbrhk\" (UID: \"4d3ed278-258b-4590-846b-c11283cbbba3\") " pod="openshift-dns/node-resolver-cbrhk"
Jul 16 04:49:59.120062 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119556    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-modprobe-d\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.120062 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119581    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-cni-bin\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.120062 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119650    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-cnibin\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.120062 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119680    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-var-lib-kubelet\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.120062 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119725    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-multus-conf-dir\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.120062 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.119752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgp8x\" (UniqueName: \"kubernetes.io/projected/dedd46bb-de3b-4d76-9276-9bc6ac152348-kube-api-access-dgp8x\") pod \"iptables-alerter-dk659\" (UID: \"dedd46bb-de3b-4d76-9276-9bc6ac152348\") " pod="openshift-network-operator/iptables-alerter-dk659"
Jul 16 04:49:59.120378 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.120116    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-node-log\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.120378 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.120172    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-run-ovn\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.121858 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.120549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72b01c84-01a4-4022-b66f-95876da5da22-env-overrides\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.121858 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.120569    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-var-lib-openvswitch\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.121858 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.120569    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-kubelet\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.121858 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.120605    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-cni-netd\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.121858 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.120607    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-run-systemd\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.121858 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.120639    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-systemd-units\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.121858 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.120696    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-run-openvswitch\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.121858 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.120737    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-etc-openvswitch\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.121858 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.121092    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2066081-de65-45f1-a46c-0f456249f49e-serviceca\") pod \"node-ca-sslfh\" (UID: \"a2066081-de65-45f1-a46c-0f456249f49e\") " pod="openshift-image-registry/node-ca-sslfh"
Jul 16 04:49:59.121858 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.121157    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-run-ovn-kubernetes\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.121858 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.121243    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-run-netns\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.121858 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.121283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72b01c84-01a4-4022-b66f-95876da5da22-host-cni-bin\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.121858 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.121403    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a3625d5-1418-40d6-b16b-1cb8a069163a-konnectivity-ca\") pod \"konnectivity-agent-5rnd9\" (UID: \"5a3625d5-1418-40d6-b16b-1cb8a069163a\") " pod="kube-system/konnectivity-agent-5rnd9"
Jul 16 04:49:59.121858 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.121513    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dedd46bb-de3b-4d76-9276-9bc6ac152348-host-slash\") pod \"iptables-alerter-dk659\" (UID: \"dedd46bb-de3b-4d76-9276-9bc6ac152348\") " pod="openshift-network-operator/iptables-alerter-dk659"
Jul 16 04:49:59.121858 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.121792    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72b01c84-01a4-4022-b66f-95876da5da22-ovnkube-script-lib\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.123821 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.122003    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72b01c84-01a4-4022-b66f-95876da5da22-ovnkube-config\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.123821 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.123134    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 04:49:59.123821 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.123155    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 04:49:59.123821 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.123169    2565 projected.go:196] Error preparing data for projected volume kube-api-access-ql6fr for pod openshift-network-diagnostics/network-check-target-pz5t7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 04:49:59.123821 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.123273    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr podName:b74a6491-0f27-40ee-8f69-b0480a1d1889 nodeName:}" failed. No retries permitted until 2026-07-16 04:49:59.623240172 +0000 UTC m=+3.039195240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ql6fr" (UniqueName: "kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr") pod "network-check-target-pz5t7" (UID: "b74a6491-0f27-40ee-8f69-b0480a1d1889") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 04:49:59.124253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.124229    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a3625d5-1418-40d6-b16b-1cb8a069163a-agent-certs\") pod \"konnectivity-agent-5rnd9\" (UID: \"5a3625d5-1418-40d6-b16b-1cb8a069163a\") " pod="kube-system/konnectivity-agent-5rnd9"
Jul 16 04:49:59.125147 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.125035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72b01c84-01a4-4022-b66f-95876da5da22-ovn-node-metrics-cert\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.127225 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.127199    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8nhc\" (UniqueName: \"kubernetes.io/projected/72b01c84-01a4-4022-b66f-95876da5da22-kube-api-access-n8nhc\") pod \"ovnkube-node-5nsc5\" (UID: \"72b01c84-01a4-4022-b66f-95876da5da22\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.128365 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.127557    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rdr9\" (UniqueName: \"kubernetes.io/projected/a2066081-de65-45f1-a46c-0f456249f49e-kube-api-access-7rdr9\") pod \"node-ca-sslfh\" (UID: \"a2066081-de65-45f1-a46c-0f456249f49e\") " pod="openshift-image-registry/node-ca-sslfh"
Jul 16 04:49:59.132003 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.131928    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgp8x\" (UniqueName: \"kubernetes.io/projected/dedd46bb-de3b-4d76-9276-9bc6ac152348-kube-api-access-dgp8x\") pod \"iptables-alerter-dk659\" (UID: \"dedd46bb-de3b-4d76-9276-9bc6ac152348\") " pod="openshift-network-operator/iptables-alerter-dk659"
Jul 16 04:49:59.141130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.141001    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal" event={"ID":"85a4e4c8efa90150c8951f2c413ffb17","Type":"ContainerStarted","Data":"b7f7b71a73a5c1d1998d538e8a76983bfe24a80598d10a86f9bb07c160ffded6"}
Jul 16 04:49:59.142853 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.142678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-199.ec2.internal" event={"ID":"4fb9095ab1b7993f0bf6be0862d6ba52","Type":"ContainerStarted","Data":"d9f2a486f491b00c516c4b9a3fee8595d929802c761338923f7a1a8cf087eecf"}
Jul 16 04:49:59.221091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221046    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-multus-cni-dir\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.221091 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-os-release\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.221338 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221148    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74612e83-5ffe-4c5d-830b-8e3f83fd8579-kubelet-config\") pod \"global-pull-secret-syncer-dmm65\" (UID: \"74612e83-5ffe-4c5d-830b-8e3f83fd8579\") " pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:49:59.221338 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221174    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x24p\" (UniqueName: \"kubernetes.io/projected/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-kube-api-access-7x24p\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.221338 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221202    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3ed278-258b-4590-846b-c11283cbbba3-tmp-dir\") pod \"node-resolver-cbrhk\" (UID: \"4d3ed278-258b-4590-846b-c11283cbbba3\") " pod="openshift-dns/node-resolver-cbrhk"
Jul 16 04:49:59.221338 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-sysctl-conf\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.221338 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221253    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf1738c9-f66f-4506-9b6e-422e9dd92071-tmp\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.221338 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221279    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.221338 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221304    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-multus-socket-dir-parent\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.221338 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvt5\" (UniqueName: \"kubernetes.io/projected/ec34c160-0104-4e3e-9e81-bd84e8266708-kube-api-access-5qvt5\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvjhg\" (UniqueName: \"kubernetes.io/projected/c028e133-a290-4758-998c-24e158389825-kube-api-access-kvjhg\") pod \"network-metrics-daemon-ldxlj\" (UID: \"c028e133-a290-4758-998c-24e158389825\") " pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-sysconfig\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221391    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-kubernetes\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-var-lib-cni-multus\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221436    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-sys-fs\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-var-lib-cni-bin\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-multus-daemon-config\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret\") pod \"global-pull-secret-syncer-dmm65\" (UID: \"74612e83-5ffe-4c5d-830b-8e3f83fd8579\") " pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221502    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs\") pod \"network-metrics-daemon-ldxlj\" (UID: \"c028e133-a290-4758-998c-24e158389825\") " pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221518    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-run\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-host\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221549    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-cni-binary-copy\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-run-k8s-cni-cncf-io\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221605    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec34c160-0104-4e3e-9e81-bd84e8266708-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-lib-modules\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-registration-dir\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.221745 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221713    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-tuned\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec34c160-0104-4e3e-9e81-bd84e8266708-cnibin\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221765    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec34c160-0104-4e3e-9e81-bd84e8266708-os-release\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-kubernetes\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.221844    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221856    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec34c160-0104-4e3e-9e81-bd84e8266708-os-release\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221873    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-multus-cni-dir\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.221922    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs podName:c028e133-a290-4758-998c-24e158389825 nodeName:}" failed. No retries permitted until 2026-07-16 04:49:59.721899936 +0000 UTC m=+3.137855007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs") pod "network-metrics-daemon-ldxlj" (UID: "c028e133-a290-4758-998c-24e158389825") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221920    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-run\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221956    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-host\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.221979    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-var-lib-cni-multus\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222015    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-os-release\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222041    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-sys-fs\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222065    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74612e83-5ffe-4c5d-830b-8e3f83fd8579-kubelet-config\") pod \"global-pull-secret-syncer-dmm65\" (UID: \"74612e83-5ffe-4c5d-830b-8e3f83fd8579\") " pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222086    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-var-lib-cni-bin\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-sysctl-conf\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec34c160-0104-4e3e-9e81-bd84e8266708-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.222408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222316    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-system-cni-dir\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222325    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-cni-binary-copy\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222337    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xx9z\" (UniqueName: \"kubernetes.io/projected/bf1738c9-f66f-4506-9b6e-422e9dd92071-kube-api-access-4xx9z\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222356    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-run-multus-certs\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222408    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec34c160-0104-4e3e-9e81-bd84e8266708-cnibin\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222562    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-run-k8s-cni-cncf-io\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222591    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-multus-daemon-config\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222605    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-multus-socket-dir-parent\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222661    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec34c160-0104-4e3e-9e81-bd84e8266708-cni-binary-copy\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-registration-dir\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222880    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-run-netns\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222895    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3ed278-258b-4590-846b-c11283cbbba3-tmp-dir\") pod \"node-resolver-cbrhk\" (UID: \"4d3ed278-258b-4590-846b-c11283cbbba3\") " pod="openshift-dns/node-resolver-cbrhk"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222910    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-hostroot\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222960    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec34c160-0104-4e3e-9e81-bd84e8266708-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.222994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-run-multus-certs\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.223031    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223036    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-run-netns\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.223422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223073    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-hostroot\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223075    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec34c160-0104-4e3e-9e81-bd84e8266708-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.223095    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret podName:74612e83-5ffe-4c5d-830b-8e3f83fd8579 nodeName:}" failed. No retries permitted until 2026-07-16 04:49:59.723078732 +0000 UTC m=+3.139033800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret") pod "global-pull-secret-syncer-dmm65" (UID: "74612e83-5ffe-4c5d-830b-8e3f83fd8579") : object "kube-system"/"original-pull-secret" not registered
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-system-cni-dir\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223145    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-sysconfig\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223195    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-socket-dir\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223223    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec34c160-0104-4e3e-9e81-bd84e8266708-cni-binary-copy\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223242    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d3ed278-258b-4590-846b-c11283cbbba3-hosts-file\") pod \"node-resolver-cbrhk\" (UID: \"4d3ed278-258b-4590-846b-c11283cbbba3\") " pod="openshift-dns/node-resolver-cbrhk"
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-modprobe-d\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d3ed278-258b-4590-846b-c11283cbbba3-hosts-file\") pod \"node-resolver-cbrhk\" (UID: \"4d3ed278-258b-4590-846b-c11283cbbba3\") " pod="openshift-dns/node-resolver-cbrhk"
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223304    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-lib-modules\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223313    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-socket-dir\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223332    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-cnibin\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223368    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-cnibin\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223371    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-var-lib-kubelet\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223339    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-modprobe-d\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223400    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-multus-conf-dir\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.224427 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223428    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-systemd\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223438    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-host-var-lib-kubelet\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223476    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-systemd\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223490    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-multus-conf-dir\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223540    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-var-lib-kubelet\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223574    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-etc-kubernetes\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223619    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kgwf\" (UniqueName: \"kubernetes.io/projected/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-kube-api-access-8kgwf\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-etc-kubernetes\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223659    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-var-lib-kubelet\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74612e83-5ffe-4c5d-830b-8e3f83fd8579-dbus\") pod \"global-pull-secret-syncer-dmm65\" (UID: \"74612e83-5ffe-4c5d-830b-8e3f83fd8579\") " pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec34c160-0104-4e3e-9e81-bd84e8266708-system-cni-dir\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223920    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhsn\" (UniqueName: \"kubernetes.io/projected/4d3ed278-258b-4590-846b-c11283cbbba3-kube-api-access-vdhsn\") pod \"node-resolver-cbrhk\" (UID: \"4d3ed278-258b-4590-846b-c11283cbbba3\") " pod="openshift-dns/node-resolver-cbrhk"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223846    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74612e83-5ffe-4c5d-830b-8e3f83fd8579-dbus\") pod \"global-pull-secret-syncer-dmm65\" (UID: \"74612e83-5ffe-4c5d-830b-8e3f83fd8579\") " pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223945    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-sysctl-d\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223978    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec34c160-0104-4e3e-9e81-bd84e8266708-system-cni-dir\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.223994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-sys\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.224025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-device-dir\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.225253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.224053    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.225969 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.224080    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec34c160-0104-4e3e-9e81-bd84e8266708-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.225969 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.224091    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-sysctl-d\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.225969 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.224104    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf1738c9-f66f-4506-9b6e-422e9dd92071-sys\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.225969 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.224190    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-device-dir\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.225969 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.224285    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.225969 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.224526    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec34c160-0104-4e3e-9e81-bd84e8266708-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.225969 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.225315    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf1738c9-f66f-4506-9b6e-422e9dd92071-tmp\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.225969 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.225714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf1738c9-f66f-4506-9b6e-422e9dd92071-etc-tuned\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.238312 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.238229    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x24p\" (UniqueName: \"kubernetes.io/projected/d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc-kube-api-access-7x24p\") pod \"aws-ebs-csi-driver-node-rs87c\" (UID: \"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.238952 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.238897    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kgwf\" (UniqueName: \"kubernetes.io/projected/ffcbf45d-b832-40f9-ab2c-075cf8b4ccac-kube-api-access-8kgwf\") pod \"multus-t8rkk\" (UID: \"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac\") " pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.238952 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.238920    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xx9z\" (UniqueName: \"kubernetes.io/projected/bf1738c9-f66f-4506-9b6e-422e9dd92071-kube-api-access-4xx9z\") pod \"tuned-w5m9p\" (UID: \"bf1738c9-f66f-4506-9b6e-422e9dd92071\") " pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.239486 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.239461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvjhg\" (UniqueName: \"kubernetes.io/projected/c028e133-a290-4758-998c-24e158389825-kube-api-access-kvjhg\") pod \"network-metrics-daemon-ldxlj\" (UID: \"c028e133-a290-4758-998c-24e158389825\") " pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:49:59.240688 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.240662    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvt5\" (UniqueName: \"kubernetes.io/projected/ec34c160-0104-4e3e-9e81-bd84e8266708-kube-api-access-5qvt5\") pod \"multus-additional-cni-plugins-9fsdp\" (UID: \"ec34c160-0104-4e3e-9e81-bd84e8266708\") " pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.240790 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.240721    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhsn\" (UniqueName: \"kubernetes.io/projected/4d3ed278-258b-4590-846b-c11283cbbba3-kube-api-access-vdhsn\") pod \"node-resolver-cbrhk\" (UID: \"4d3ed278-258b-4590-846b-c11283cbbba3\") " pod="openshift-dns/node-resolver-cbrhk"
Jul 16 04:49:59.262405 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.262372    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 04:49:59.309084 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.308801    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5rnd9"
Jul 16 04:49:59.327752 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.327457    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sslfh"
Jul 16 04:49:59.340798 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.340762    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dk659"
Jul 16 04:49:59.353370 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.353326    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:49:59.362164 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.362127    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c"
Jul 16 04:49:59.369774 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.369741    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9fsdp"
Jul 16 04:49:59.376259 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.376235    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w5m9p"
Jul 16 04:49:59.386727 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.386485    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cbrhk"
Jul 16 04:49:59.394186 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.394160    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t8rkk"
Jul 16 04:49:59.479545 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:59.479469    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec34c160_0104_4e3e_9e81_bd84e8266708.slice/crio-aa43966be7490847014d4e4c12bfbcba6324ad483c068efe567e72cbd421b694 WatchSource:0}: Error finding container aa43966be7490847014d4e4c12bfbcba6324ad483c068efe567e72cbd421b694: Status 404 returned error can't find the container with id aa43966be7490847014d4e4c12bfbcba6324ad483c068efe567e72cbd421b694
Jul 16 04:49:59.480990 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:59.480951    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3ed278_258b_4590_846b_c11283cbbba3.slice/crio-592e3d1725c68a7aa1aaf9bcaa6c2b33fcf5b54dea5be5a5b8e4b3abf173abef WatchSource:0}: Error finding container 592e3d1725c68a7aa1aaf9bcaa6c2b33fcf5b54dea5be5a5b8e4b3abf173abef: Status 404 returned error can't find the container with id 592e3d1725c68a7aa1aaf9bcaa6c2b33fcf5b54dea5be5a5b8e4b3abf173abef
Jul 16 04:49:59.483395 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:59.483362    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcbf45d_b832_40f9_ab2c_075cf8b4ccac.slice/crio-888d64b02b55171be8ba341ce4612d2f1d38ef40e1320a462cd4071c87a155a0 WatchSource:0}: Error finding container 888d64b02b55171be8ba341ce4612d2f1d38ef40e1320a462cd4071c87a155a0: Status 404 returned error can't find the container with id 888d64b02b55171be8ba341ce4612d2f1d38ef40e1320a462cd4071c87a155a0
Jul 16 04:49:59.486255 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:59.486224    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1738c9_f66f_4506_9b6e_422e9dd92071.slice/crio-8f7fa2df891f03272e66e4b3a3492225b1be4177ec141adccfd4cc00902f0fa9 WatchSource:0}: Error finding container 8f7fa2df891f03272e66e4b3a3492225b1be4177ec141adccfd4cc00902f0fa9: Status 404 returned error can't find the container with id 8f7fa2df891f03272e66e4b3a3492225b1be4177ec141adccfd4cc00902f0fa9
Jul 16 04:49:59.487182 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:59.487159    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ff82ea_2ff9_44d0_b14f_2b966e84b6cc.slice/crio-9007380d204c28195cabd5fba31a513c1f0f04496776fecf597c74c871199fcb WatchSource:0}: Error finding container 9007380d204c28195cabd5fba31a513c1f0f04496776fecf597c74c871199fcb: Status 404 returned error can't find the container with id 9007380d204c28195cabd5fba31a513c1f0f04496776fecf597c74c871199fcb
Jul 16 04:49:59.488469 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:59.488378    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2066081_de65_45f1_a46c_0f456249f49e.slice/crio-ae2a61ccd654e48b3bd1a841279a1eb28bcc3eb15c433b03771ddf19a80d9efc WatchSource:0}: Error finding container ae2a61ccd654e48b3bd1a841279a1eb28bcc3eb15c433b03771ddf19a80d9efc: Status 404 returned error can't find the container with id ae2a61ccd654e48b3bd1a841279a1eb28bcc3eb15c433b03771ddf19a80d9efc
Jul 16 04:49:59.489130 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:59.489103    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b01c84_01a4_4022_b66f_95876da5da22.slice/crio-a6261c6392361dc2834450c9fb7b9868967587bd1d878d96e63e70c8e74ad74d WatchSource:0}: Error finding container a6261c6392361dc2834450c9fb7b9868967587bd1d878d96e63e70c8e74ad74d: Status 404 returned error can't find the container with id a6261c6392361dc2834450c9fb7b9868967587bd1d878d96e63e70c8e74ad74d
Jul 16 04:49:59.490849 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:59.490775    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3625d5_1418_40d6_b16b_1cb8a069163a.slice/crio-bf130f7613addde54a2ec2610f4e348b2076de34816ae2dce94757b6263a02da WatchSource:0}: Error finding container bf130f7613addde54a2ec2610f4e348b2076de34816ae2dce94757b6263a02da: Status 404 returned error can't find the container with id bf130f7613addde54a2ec2610f4e348b2076de34816ae2dce94757b6263a02da
Jul 16 04:49:59.491072 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:49:59.491013    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddedd46bb_de3b_4d76_9276_9bc6ac152348.slice/crio-58ae8a67076fc24f4f132eee40c5b9cda7ebc0e5c094566f609d9bc813a283a7 WatchSource:0}: Error finding container 58ae8a67076fc24f4f132eee40c5b9cda7ebc0e5c094566f609d9bc813a283a7: Status 404 returned error can't find the container with id 58ae8a67076fc24f4f132eee40c5b9cda7ebc0e5c094566f609d9bc813a283a7
Jul 16 04:49:59.627459 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.627420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6fr\" (UniqueName: \"kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr\") pod \"network-check-target-pz5t7\" (UID: \"b74a6491-0f27-40ee-8f69-b0480a1d1889\") " pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:49:59.627679 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.627586    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 04:49:59.627679 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.627608    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 04:49:59.627679 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.627622    2565 projected.go:196] Error preparing data for projected volume kube-api-access-ql6fr for pod openshift-network-diagnostics/network-check-target-pz5t7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 04:49:59.627889 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.627709    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr podName:b74a6491-0f27-40ee-8f69-b0480a1d1889 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:00.627687719 +0000 UTC m=+4.043642772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ql6fr" (UniqueName: "kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr") pod "network-check-target-pz5t7" (UID: "b74a6491-0f27-40ee-8f69-b0480a1d1889") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 04:49:59.728656 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.728619    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret\") pod \"global-pull-secret-syncer-dmm65\" (UID: \"74612e83-5ffe-4c5d-830b-8e3f83fd8579\") " pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:49:59.728819 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:49:59.728674    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs\") pod \"network-metrics-daemon-ldxlj\" (UID: \"c028e133-a290-4758-998c-24e158389825\") " pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:49:59.728819 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.728784    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 04:49:59.728986 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.728836    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 04:49:59.728986 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.728854    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs podName:c028e133-a290-4758-998c-24e158389825 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:00.728833052 +0000 UTC m=+4.144788130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs") pod "network-metrics-daemon-ldxlj" (UID: "c028e133-a290-4758-998c-24e158389825") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 04:49:59.728986 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:49:59.728921    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret podName:74612e83-5ffe-4c5d-830b-8e3f83fd8579 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:00.72890226 +0000 UTC m=+4.144857327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret") pod "global-pull-secret-syncer-dmm65" (UID: "74612e83-5ffe-4c5d-830b-8e3f83fd8579") : object "kube-system"/"original-pull-secret" not registered
Jul 16 04:50:00.055573 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.055476    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 04:44:59 +0000 UTC" deadline="2028-03-30 18:35:54.150310692 +0000 UTC"
Jul 16 04:50:00.055573 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.055524    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14965h45m54.094791826s"
Jul 16 04:50:00.135440 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.135401    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:00.135603 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:00.135555    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:00.148844 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.148691    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dk659" event={"ID":"dedd46bb-de3b-4d76-9276-9bc6ac152348","Type":"ContainerStarted","Data":"58ae8a67076fc24f4f132eee40c5b9cda7ebc0e5c094566f609d9bc813a283a7"}
Jul 16 04:50:00.152019 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.151974    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5rnd9" event={"ID":"5a3625d5-1418-40d6-b16b-1cb8a069163a","Type":"ContainerStarted","Data":"bf130f7613addde54a2ec2610f4e348b2076de34816ae2dce94757b6263a02da"}
Jul 16 04:50:00.154198 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.154165    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sslfh" event={"ID":"a2066081-de65-45f1-a46c-0f456249f49e","Type":"ContainerStarted","Data":"ae2a61ccd654e48b3bd1a841279a1eb28bcc3eb15c433b03771ddf19a80d9efc"}
Jul 16 04:50:00.164656 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.164418    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c" event={"ID":"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc","Type":"ContainerStarted","Data":"9007380d204c28195cabd5fba31a513c1f0f04496776fecf597c74c871199fcb"}
Jul 16 04:50:00.168143 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.168120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w5m9p" event={"ID":"bf1738c9-f66f-4506-9b6e-422e9dd92071","Type":"ContainerStarted","Data":"8f7fa2df891f03272e66e4b3a3492225b1be4177ec141adccfd4cc00902f0fa9"}
Jul 16 04:50:00.169943 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.169870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5" event={"ID":"72b01c84-01a4-4022-b66f-95876da5da22","Type":"ContainerStarted","Data":"a6261c6392361dc2834450c9fb7b9868967587bd1d878d96e63e70c8e74ad74d"}
Jul 16 04:50:00.173276 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.173250    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t8rkk" event={"ID":"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac","Type":"ContainerStarted","Data":"888d64b02b55171be8ba341ce4612d2f1d38ef40e1320a462cd4071c87a155a0"}
Jul 16 04:50:00.177739 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.177685    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cbrhk" event={"ID":"4d3ed278-258b-4590-846b-c11283cbbba3","Type":"ContainerStarted","Data":"592e3d1725c68a7aa1aaf9bcaa6c2b33fcf5b54dea5be5a5b8e4b3abf173abef"}
Jul 16 04:50:00.180466 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.180390    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fsdp" event={"ID":"ec34c160-0104-4e3e-9e81-bd84e8266708","Type":"ContainerStarted","Data":"aa43966be7490847014d4e4c12bfbcba6324ad483c068efe567e72cbd421b694"}
Jul 16 04:50:00.186147 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.186117    2565 generic.go:356] "Generic (PLEG): container finished" podID="85a4e4c8efa90150c8951f2c413ffb17" containerID="e538fa24d5142138c46af0ee48e58d8813f1a0429a590b4c036531c32e3aa790" exitCode=0
Jul 16 04:50:00.186912 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.186300    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal" event={"ID":"85a4e4c8efa90150c8951f2c413ffb17","Type":"ContainerDied","Data":"e538fa24d5142138c46af0ee48e58d8813f1a0429a590b4c036531c32e3aa790"}
Jul 16 04:50:00.636901 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.636857    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6fr\" (UniqueName: \"kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr\") pod \"network-check-target-pz5t7\" (UID: \"b74a6491-0f27-40ee-8f69-b0480a1d1889\") " pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:00.637135 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:00.637115    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 04:50:00.637203 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:00.637148    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 04:50:00.637203 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:00.637161    2565 projected.go:196] Error preparing data for projected volume kube-api-access-ql6fr for pod openshift-network-diagnostics/network-check-target-pz5t7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 04:50:00.637327 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:00.637225    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr podName:b74a6491-0f27-40ee-8f69-b0480a1d1889 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:02.637205524 +0000 UTC m=+6.053160584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ql6fr" (UniqueName: "kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr") pod "network-check-target-pz5t7" (UID: "b74a6491-0f27-40ee-8f69-b0480a1d1889") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 04:50:00.738435 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.738329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret\") pod \"global-pull-secret-syncer-dmm65\" (UID: \"74612e83-5ffe-4c5d-830b-8e3f83fd8579\") " pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:00.738435 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:00.738393    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs\") pod \"network-metrics-daemon-ldxlj\" (UID: \"c028e133-a290-4758-998c-24e158389825\") " pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:00.738710 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:00.738594    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 04:50:00.738710 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:00.738685    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs podName:c028e133-a290-4758-998c-24e158389825 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:02.73866041 +0000 UTC m=+6.154615480 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs") pod "network-metrics-daemon-ldxlj" (UID: "c028e133-a290-4758-998c-24e158389825") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 04:50:00.738835 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:00.738773    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 04:50:00.738835 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:00.738818    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret podName:74612e83-5ffe-4c5d-830b-8e3f83fd8579 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:02.738804622 +0000 UTC m=+6.154759697 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret") pod "global-pull-secret-syncer-dmm65" (UID: "74612e83-5ffe-4c5d-830b-8e3f83fd8579") : object "kube-system"/"original-pull-secret" not registered
Jul 16 04:50:01.056356 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:01.056227    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 04:44:59 +0000 UTC" deadline="2028-03-15 05:32:22.668641894 +0000 UTC"
Jul 16 04:50:01.056356 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:01.056299    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14592h42m21.61234775s"
Jul 16 04:50:01.138258 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:01.138215    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:01.138458 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:01.138357    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:01.138458 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:01.138446    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:01.138587 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:01.138534    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:01.204803 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:01.204751    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-199.ec2.internal" event={"ID":"4fb9095ab1b7993f0bf6be0862d6ba52","Type":"ContainerStarted","Data":"dfd48152a52dce9f345227461a353394df1ea64dcbae6e8c65ff578ba3d0a06f"}
Jul 16 04:50:01.223954 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:01.223912    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal" event={"ID":"85a4e4c8efa90150c8951f2c413ffb17","Type":"ContainerStarted","Data":"a324cd322619571b8885eefb7d86c4107abac3e439ab6aa94fdb0ccbfbd74345"}
Jul 16 04:50:01.241413 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:01.241259    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-199.ec2.internal" podStartSLOduration=2.241235244 podStartE2EDuration="2.241235244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:50:01.241009698 +0000 UTC m=+4.656964777" watchObservedRunningTime="2026-07-16 04:50:01.241235244 +0000 UTC m=+4.657190324"
Jul 16 04:50:01.242911 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:01.242117    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-199.ec2.internal" podStartSLOduration=2.24208983 podStartE2EDuration="2.24208983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:50:01.219541992 +0000 UTC m=+4.635497072" watchObservedRunningTime="2026-07-16 04:50:01.24208983 +0000 UTC m=+4.658044908"
Jul 16 04:50:02.138933 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:02.138890    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:02.139471 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:02.139040    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:02.658206 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:02.658147    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6fr\" (UniqueName: \"kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr\") pod \"network-check-target-pz5t7\" (UID: \"b74a6491-0f27-40ee-8f69-b0480a1d1889\") " pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:02.658416 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:02.658377    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 04:50:02.658516 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:02.658421    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 04:50:02.658516 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:02.658436    2565 projected.go:196] Error preparing data for projected volume kube-api-access-ql6fr for pod openshift-network-diagnostics/network-check-target-pz5t7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 04:50:02.658516 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:02.658499    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr podName:b74a6491-0f27-40ee-8f69-b0480a1d1889 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:06.658479373 +0000 UTC m=+10.074434428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ql6fr" (UniqueName: "kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr") pod "network-check-target-pz5t7" (UID: "b74a6491-0f27-40ee-8f69-b0480a1d1889") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 04:50:02.759688 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:02.759416    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret\") pod \"global-pull-secret-syncer-dmm65\" (UID: \"74612e83-5ffe-4c5d-830b-8e3f83fd8579\") " pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:02.759688 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:02.759471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs\") pod \"network-metrics-daemon-ldxlj\" (UID: \"c028e133-a290-4758-998c-24e158389825\") " pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:02.759688 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:02.759621    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 04:50:02.760007 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:02.759711    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs podName:c028e133-a290-4758-998c-24e158389825 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:06.759689215 +0000 UTC m=+10.175644272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs") pod "network-metrics-daemon-ldxlj" (UID: "c028e133-a290-4758-998c-24e158389825") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 04:50:02.760373 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:02.760276    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 04:50:02.760373 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:02.760334    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret podName:74612e83-5ffe-4c5d-830b-8e3f83fd8579 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:06.760316472 +0000 UTC m=+10.176271530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret") pod "global-pull-secret-syncer-dmm65" (UID: "74612e83-5ffe-4c5d-830b-8e3f83fd8579") : object "kube-system"/"original-pull-secret" not registered
Jul 16 04:50:03.138913 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:03.138827    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:03.139074 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:03.138982    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:03.139074 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:03.139050    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:03.139529 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:03.139139    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:04.136325 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:04.136280    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:04.136521 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:04.136430    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:05.136856 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:05.136765    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:05.137321 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:05.136920    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:05.137321 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:05.137279    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:05.137447 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:05.137398    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:06.136351 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:06.136312    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:06.136559 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:06.136450    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:06.696496 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:06.696450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6fr\" (UniqueName: \"kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr\") pod \"network-check-target-pz5t7\" (UID: \"b74a6491-0f27-40ee-8f69-b0480a1d1889\") " pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:06.696941 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:06.696692    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 04:50:06.696941 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:06.696723    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 04:50:06.696941 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:06.696734    2565 projected.go:196] Error preparing data for projected volume kube-api-access-ql6fr for pod openshift-network-diagnostics/network-check-target-pz5t7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 04:50:06.696941 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:06.696801    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr podName:b74a6491-0f27-40ee-8f69-b0480a1d1889 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:14.696777729 +0000 UTC m=+18.112732787 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ql6fr" (UniqueName: "kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr") pod "network-check-target-pz5t7" (UID: "b74a6491-0f27-40ee-8f69-b0480a1d1889") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 04:50:06.797662 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:06.797596    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret\") pod \"global-pull-secret-syncer-dmm65\" (UID: \"74612e83-5ffe-4c5d-830b-8e3f83fd8579\") " pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:06.797841 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:06.797684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs\") pod \"network-metrics-daemon-ldxlj\" (UID: \"c028e133-a290-4758-998c-24e158389825\") " pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:06.797841 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:06.797730    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 04:50:06.797841 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:06.797801    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret podName:74612e83-5ffe-4c5d-830b-8e3f83fd8579 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:14.7977834 +0000 UTC m=+18.213738456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret") pod "global-pull-secret-syncer-dmm65" (UID: "74612e83-5ffe-4c5d-830b-8e3f83fd8579") : object "kube-system"/"original-pull-secret" not registered
Jul 16 04:50:06.798038 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:06.797858    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 04:50:06.798038 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:06.797914    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs podName:c028e133-a290-4758-998c-24e158389825 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:14.797895964 +0000 UTC m=+18.213851031 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs") pod "network-metrics-daemon-ldxlj" (UID: "c028e133-a290-4758-998c-24e158389825") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 04:50:07.136207 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:07.136111    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:07.136391 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:07.136205    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:07.136391 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:07.136226    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:07.136391 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:07.136337    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:08.135863 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:08.135826    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:08.136362 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:08.135958    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:09.137570 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:09.137480    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:09.137570 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:09.137540    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:09.138075 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:09.137640    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:09.138075 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:09.137797    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:10.136070 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:10.136032    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:10.136298 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:10.136175    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:11.135941 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:11.135903    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:11.135941 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:11.135942    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:11.136505 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:11.136048    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:11.136505 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:11.136189    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:12.136025 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:12.135990    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:12.136527 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:12.136111    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:13.135476 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:13.135443    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:13.135662 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:13.135618    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:13.135729 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:13.135659    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:13.136299 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:13.136109    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:14.136116 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:14.136076    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:14.136327 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:14.136229    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:14.756567 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:14.756531    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6fr\" (UniqueName: \"kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr\") pod \"network-check-target-pz5t7\" (UID: \"b74a6491-0f27-40ee-8f69-b0480a1d1889\") " pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:14.756885 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:14.756709    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 04:50:14.756885 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:14.756729    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 04:50:14.756885 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:14.756744    2565 projected.go:196] Error preparing data for projected volume kube-api-access-ql6fr for pod openshift-network-diagnostics/network-check-target-pz5t7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 04:50:14.756885 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:14.756800    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr podName:b74a6491-0f27-40ee-8f69-b0480a1d1889 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:30.756786415 +0000 UTC m=+34.172741482 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ql6fr" (UniqueName: "kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr") pod "network-check-target-pz5t7" (UID: "b74a6491-0f27-40ee-8f69-b0480a1d1889") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 04:50:14.856996 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:14.856960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret\") pod \"global-pull-secret-syncer-dmm65\" (UID: \"74612e83-5ffe-4c5d-830b-8e3f83fd8579\") " pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:14.856996 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:14.857000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs\") pod \"network-metrics-daemon-ldxlj\" (UID: \"c028e133-a290-4758-998c-24e158389825\") " pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:14.857255 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:14.857124    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 04:50:14.857255 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:14.857124    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 04:50:14.857255 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:14.857190    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret podName:74612e83-5ffe-4c5d-830b-8e3f83fd8579 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:30.857174843 +0000 UTC m=+34.273129897 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret") pod "global-pull-secret-syncer-dmm65" (UID: "74612e83-5ffe-4c5d-830b-8e3f83fd8579") : object "kube-system"/"original-pull-secret" not registered
Jul 16 04:50:14.857255 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:14.857208    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs podName:c028e133-a290-4758-998c-24e158389825 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:30.857200928 +0000 UTC m=+34.273155980 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs") pod "network-metrics-daemon-ldxlj" (UID: "c028e133-a290-4758-998c-24e158389825") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 04:50:15.136212 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:15.136115    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:15.136212 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:15.136143    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:15.136718 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:15.136271    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:15.136718 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:15.136409    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:16.135388 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:16.135348    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:16.135586 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:16.135473    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:17.137668 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:17.137146    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:17.137668 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:17.137277    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:17.137668 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:17.137296    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:17.137668 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:17.137414    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:18.136034 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:18.135994    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:18.136236 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:18.136126    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:19.136072 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:19.136034    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:19.136686 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:19.136185    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:19.136686 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:19.136535    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:19.136686 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:19.136662    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:20.135822 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:20.135793    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:20.136321 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:20.135932    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:20.281498 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:20.280632    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w5m9p" event={"ID":"bf1738c9-f66f-4506-9b6e-422e9dd92071","Type":"ContainerStarted","Data":"5e0ded9eff494d50ae427a9d9f6e9752da12781b1696976ec97b5c20ce389289"}
Jul 16 04:50:20.308067 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:20.306883    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w5m9p" podStartSLOduration=1.698189559 podStartE2EDuration="22.306861063s" totalImagesPullingTime="20.608671504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:50:20.305856871 +0000 UTC m=+23.721811955" watchObservedRunningTime="2026-07-16 04:50:20.306861063 +0000 UTC m=+23.722816137"
Jul 16 04:50:21.135761 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.135450    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:21.135958 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.135512    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:21.135958 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:21.135829    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:21.135958 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:21.135902    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:21.290354 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.290314    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5rnd9" event={"ID":"5a3625d5-1418-40d6-b16b-1cb8a069163a","Type":"ContainerStarted","Data":"b1ad13579a66fa80e7422b8dcf6ff7aadc74a595b446b0db957e566b08b421c0"}
Jul 16 04:50:21.291948 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.291918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sslfh" event={"ID":"a2066081-de65-45f1-a46c-0f456249f49e","Type":"ContainerStarted","Data":"905a243cd68a82afe3abcb38d10a7bdb14f6f6b0c7337f71c51d3a4bc996c163"}
Jul 16 04:50:21.293380 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.293352    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c" event={"ID":"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc","Type":"ContainerStarted","Data":"8589f0baf0c3abcc9a718c62482ae73661d1efb4bcede218eb3815a6f39b9d37"}
Jul 16 04:50:21.296062 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.296041    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 04:50:21.296388 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.296369    2565 generic.go:356] "Generic (PLEG): container finished" podID="72b01c84-01a4-4022-b66f-95876da5da22" containerID="d3da774c56179b1174e28ab56b53aa05c277810d325b53370e7b3466257fbd0a" exitCode=1
Jul 16 04:50:21.296466 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.296434    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5" event={"ID":"72b01c84-01a4-4022-b66f-95876da5da22","Type":"ContainerStarted","Data":"04c56044cab5682fdd183070076e6641ad09502951c6b8c8e827d368a53858da"}
Jul 16 04:50:21.296520 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.296499    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5" event={"ID":"72b01c84-01a4-4022-b66f-95876da5da22","Type":"ContainerStarted","Data":"2055e2e61322cb7d989ef93fa1633fe788eeee52237c447714cc48a0ee737ad1"}
Jul 16 04:50:21.296566 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.296549    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5" event={"ID":"72b01c84-01a4-4022-b66f-95876da5da22","Type":"ContainerStarted","Data":"8ebfc2daa5bc1f1622eb9d0f8cb5857b27264aaf4c0cec6c6a05ebdebe5899e3"}
Jul 16 04:50:21.296630 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.296578    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5" event={"ID":"72b01c84-01a4-4022-b66f-95876da5da22","Type":"ContainerStarted","Data":"d532e7941a982c9306a5c44dcff30ca4c89d92fa2d274e820442163029da9d4f"}
Jul 16 04:50:21.296684 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.296631    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5" event={"ID":"72b01c84-01a4-4022-b66f-95876da5da22","Type":"ContainerStarted","Data":"946a42a2f4ca8aa7a0e22b4b1c5daedd1a71ce5b3b9b9e131a4b18849bd468e1"}
Jul 16 04:50:21.296684 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.296679    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5" event={"ID":"72b01c84-01a4-4022-b66f-95876da5da22","Type":"ContainerDied","Data":"d3da774c56179b1174e28ab56b53aa05c277810d325b53370e7b3466257fbd0a"}
Jul 16 04:50:21.297871 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.297851    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t8rkk" event={"ID":"ffcbf45d-b832-40f9-ab2c-075cf8b4ccac","Type":"ContainerStarted","Data":"75dc62394276104d8a0775c4ef992c101d5434320bfab5f4802dfb5fa7ac9533"}
Jul 16 04:50:21.299283 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.299265    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cbrhk" event={"ID":"4d3ed278-258b-4590-846b-c11283cbbba3","Type":"ContainerStarted","Data":"72f3355b469a95c5b52a926e13032a89814a30a44aa47053d1155a4b2b84cb87"}
Jul 16 04:50:21.300569 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.300550    2565 generic.go:356] "Generic (PLEG): container finished" podID="ec34c160-0104-4e3e-9e81-bd84e8266708" containerID="9061d159fcd9b09c9bf3d979d39e315856c4fe1b63ff9dceecc6e58b1c24cdec" exitCode=0
Jul 16 04:50:21.300672 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.300578    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fsdp" event={"ID":"ec34c160-0104-4e3e-9e81-bd84e8266708","Type":"ContainerDied","Data":"9061d159fcd9b09c9bf3d979d39e315856c4fe1b63ff9dceecc6e58b1c24cdec"}
Jul 16 04:50:21.304132 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.304079    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5rnd9" podStartSLOduration=2.723372103 podStartE2EDuration="23.304061056s" totalImagesPullingTime="20.580688953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:50:21.303342274 +0000 UTC m=+24.719297349" watchObservedRunningTime="2026-07-16 04:50:21.304061056 +0000 UTC m=+24.720016130"
Jul 16 04:50:21.334845 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.334792    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cbrhk" podStartSLOduration=2.724156563 podStartE2EDuration="23.334776685s" totalImagesPullingTime="20.610620122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:50:21.334273584 +0000 UTC m=+24.750228657" watchObservedRunningTime="2026-07-16 04:50:21.334776685 +0000 UTC m=+24.750731757"
Jul 16 04:50:21.346899 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.346846    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sslfh" podStartSLOduration=11.343264306 podStartE2EDuration="23.346836343s" totalImagesPullingTime="12.003572037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:50:21.346480379 +0000 UTC m=+24.762435453" watchObservedRunningTime="2026-07-16 04:50:21.346836343 +0000 UTC m=+24.762791416"
Jul 16 04:50:21.362827 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.362773    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-t8rkk" podStartSLOduration=2.688882735 podStartE2EDuration="23.36275601s" totalImagesPullingTime="20.673873275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:50:21.362094445 +0000 UTC m=+24.778049517" watchObservedRunningTime="2026-07-16 04:50:21.36275601 +0000 UTC m=+24.778711080"
Jul 16 04:50:21.701200 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:21.701163    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 04:50:22.078411 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:22.078311    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T04:50:21.701188316Z","UUID":"5fa9419a-fb28-4c2f-9e89-de5c34e1dc08","Handler":null,"Name":"","Endpoint":""}
Jul 16 04:50:22.079750 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:22.079713    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 04:50:22.079874 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:22.079761    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 04:50:22.135533 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:22.135499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:22.135809 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:22.135644    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:22.305061 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:22.305025    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dk659" event={"ID":"dedd46bb-de3b-4d76-9276-9bc6ac152348","Type":"ContainerStarted","Data":"3e8343d3537da091ab65f23027589361babf3e3aba46db87428da9a75b8491de"}
Jul 16 04:50:22.308256 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:22.308222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c" event={"ID":"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc","Type":"ContainerStarted","Data":"5e65fdf11b773343291508d1069ced57417b2ec54b3c967ab4376f05609ecc2b"}
Jul 16 04:50:23.135905 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:23.135610    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:23.136114 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:23.135669    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:23.136114 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:23.135966    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:23.136114 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:23.136052    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:23.312330 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:23.312290    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c" event={"ID":"d4ff82ea-2ff9-44d0-b14f-2b966e84b6cc","Type":"ContainerStarted","Data":"fdad6d29e9953fad9fbd17a84849060165f47425f4dc1c1fb6046367c734d948"}
Jul 16 04:50:23.316178 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:23.316153    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 04:50:23.316662 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:23.316633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5" event={"ID":"72b01c84-01a4-4022-b66f-95876da5da22","Type":"ContainerStarted","Data":"0a485e837f9bb06d8664be8de63b40d530c0c37c3d3b2cac5610c24094c56ef1"}
Jul 16 04:50:23.329455 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:23.329359    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dk659" podStartSLOduration=9.574265702 podStartE2EDuration="25.329337697s" totalImagesPullingTime="15.755071995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:50:22.320672658 +0000 UTC m=+25.736627730" watchObservedRunningTime="2026-07-16 04:50:23.329337697 +0000 UTC m=+26.745292837"
Jul 16 04:50:23.329984 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:23.329946    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rs87c" podStartSLOduration=2.446251903 podStartE2EDuration="25.329929311s" totalImagesPullingTime="22.883677408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:49:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:50:23.327482844 +0000 UTC m=+26.743437920" watchObservedRunningTime="2026-07-16 04:50:23.329929311 +0000 UTC m=+26.745884385"
Jul 16 04:50:24.135552 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:24.135516    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:24.135770 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:24.135665    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:25.136185 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:25.136151    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:25.136867 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:25.136151    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:25.136867 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:25.136287    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:25.136867 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:25.136351    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:25.438483 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:25.438455    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5rnd9"
Jul 16 04:50:25.439212 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:25.439193    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5rnd9"
Jul 16 04:50:26.135908 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:26.135673    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:26.136061 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:26.135938    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:26.328179 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:26.328155    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 04:50:26.328911 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:26.328519    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5" event={"ID":"72b01c84-01a4-4022-b66f-95876da5da22","Type":"ContainerStarted","Data":"4fd3df12fed18777a9cb2335ce8efba27bc9003b6a56b25205dfbd17693518fc"}
Jul 16 04:50:26.328911 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:26.328810    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:50:26.329089 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:26.329043    2565 scope.go:117] "RemoveContainer" containerID="d3da774c56179b1174e28ab56b53aa05c277810d325b53370e7b3466257fbd0a"
Jul 16 04:50:26.330581 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:26.330558    2565 generic.go:356] "Generic (PLEG): container finished" podID="ec34c160-0104-4e3e-9e81-bd84e8266708" containerID="e4bac4e2c9d201f093e0027d0a0b28d6ec81d27cf4954b7751bf5a2d900901c8" exitCode=0
Jul 16 04:50:26.330717 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:26.330638    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fsdp" event={"ID":"ec34c160-0104-4e3e-9e81-bd84e8266708","Type":"ContainerDied","Data":"e4bac4e2c9d201f093e0027d0a0b28d6ec81d27cf4954b7751bf5a2d900901c8"}
Jul 16 04:50:26.330906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:26.330886    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5rnd9"
Jul 16 04:50:26.331475 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:26.331448    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5rnd9"
Jul 16 04:50:26.347449 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:26.347415    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:50:27.137332 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:27.137248    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:27.137510 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:27.137377    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:27.137510 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:27.137416    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:27.137667 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:27.137520    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:27.337658 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:27.337636    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 04:50:27.338066 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:27.338041    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5" event={"ID":"72b01c84-01a4-4022-b66f-95876da5da22","Type":"ContainerStarted","Data":"e459eda340781a4c77634b03c366424b925cd1b656b62e12d5df4c85026c8fa6"}
Jul 16 04:50:27.341725 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:27.339376    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:50:27.341725 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:27.339500    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:50:27.359971 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:27.359814    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:50:27.367853 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:27.367776    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5" podStartSLOduration=8.720286732 podStartE2EDuration="29.367759567s" totalImagesPullingTime="20.647472835s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:50:27.367101748 +0000 UTC m=+30.783056824" watchObservedRunningTime="2026-07-16 04:50:27.367759567 +0000 UTC m=+30.783714640"
Jul 16 04:50:27.674314 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:27.674235    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dmm65"]
Jul 16 04:50:27.674473 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:27.674348    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:27.674473 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:27.674431    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:27.677983 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:27.677940    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ldxlj"]
Jul 16 04:50:27.678150 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:27.678055    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:27.678221 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:27.678195    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:27.678477 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:27.678455    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pz5t7"]
Jul 16 04:50:27.678560 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:27.678548    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:27.678663 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:27.678645    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:28.342388 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:28.342349    2565 generic.go:356] "Generic (PLEG): container finished" podID="ec34c160-0104-4e3e-9e81-bd84e8266708" containerID="a6ec1040d7cb024eab60d603564085e4f212295017e928816ac9900063ad3844" exitCode=0
Jul 16 04:50:28.342894 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:28.342407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fsdp" event={"ID":"ec34c160-0104-4e3e-9e81-bd84e8266708","Type":"ContainerDied","Data":"a6ec1040d7cb024eab60d603564085e4f212295017e928816ac9900063ad3844"}
Jul 16 04:50:29.135726 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:29.135696    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:29.135726 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:29.135714    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:29.135945 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:29.135696    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:29.135945 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:29.135833    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:29.135945 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:29.135920    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:29.136095 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:29.135992    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:30.350064 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:30.350028    2565 generic.go:356] "Generic (PLEG): container finished" podID="ec34c160-0104-4e3e-9e81-bd84e8266708" containerID="ee53d8f573802ed944575b26771561f553fccdd3c6979fc667b634641e0abe2f" exitCode=0
Jul 16 04:50:30.350739 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:30.350087    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fsdp" event={"ID":"ec34c160-0104-4e3e-9e81-bd84e8266708","Type":"ContainerDied","Data":"ee53d8f573802ed944575b26771561f553fccdd3c6979fc667b634641e0abe2f"}
Jul 16 04:50:30.783408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:30.783311    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6fr\" (UniqueName: \"kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr\") pod \"network-check-target-pz5t7\" (UID: \"b74a6491-0f27-40ee-8f69-b0480a1d1889\") " pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:30.783612 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:30.783500    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 04:50:30.783612 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:30.783523    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 04:50:30.783612 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:30.783536    2565 projected.go:196] Error preparing data for projected volume kube-api-access-ql6fr for pod openshift-network-diagnostics/network-check-target-pz5t7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 04:50:30.783769 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:30.783627    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr podName:b74a6491-0f27-40ee-8f69-b0480a1d1889 nodeName:}" failed. No retries permitted until 2026-07-16 04:51:02.783583639 +0000 UTC m=+66.199538704 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ql6fr" (UniqueName: "kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr") pod "network-check-target-pz5t7" (UID: "b74a6491-0f27-40ee-8f69-b0480a1d1889") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 04:50:30.884094 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:30.884054    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret\") pod \"global-pull-secret-syncer-dmm65\" (UID: \"74612e83-5ffe-4c5d-830b-8e3f83fd8579\") " pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:30.884094 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:30.884102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs\") pod \"network-metrics-daemon-ldxlj\" (UID: \"c028e133-a290-4758-998c-24e158389825\") " pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:30.884398 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:30.884216    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 04:50:30.884398 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:30.884216    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 04:50:30.884398 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:30.884277    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs podName:c028e133-a290-4758-998c-24e158389825 nodeName:}" failed. No retries permitted until 2026-07-16 04:51:02.884262254 +0000 UTC m=+66.300217306 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs") pod "network-metrics-daemon-ldxlj" (UID: "c028e133-a290-4758-998c-24e158389825") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 04:50:30.884398 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:30.884290    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret podName:74612e83-5ffe-4c5d-830b-8e3f83fd8579 nodeName:}" failed. No retries permitted until 2026-07-16 04:51:02.884284756 +0000 UTC m=+66.300239806 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret") pod "global-pull-secret-syncer-dmm65" (UID: "74612e83-5ffe-4c5d-830b-8e3f83fd8579") : object "kube-system"/"original-pull-secret" not registered
Jul 16 04:50:31.135971 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:31.135876    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:31.135971 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:31.135913    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:31.136197 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:31.135886    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:31.136197 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:31.136017    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:50:31.136197 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:31.136102    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pz5t7" podUID="b74a6491-0f27-40ee-8f69-b0480a1d1889"
Jul 16 04:50:31.136347 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:31.136200    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmm65" podUID="74612e83-5ffe-4c5d-830b-8e3f83fd8579"
Jul 16 04:50:32.862558 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.862525    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-199.ec2.internal" event="NodeReady"
Jul 16 04:50:32.863055 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.862690    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 04:50:32.898726 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.898581    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-776bb7cd46-j6xbd"]
Jul 16 04:50:32.902676 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.902643    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:32.905789 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.905760    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 04:50:32.905962 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.905819    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8whht\""
Jul 16 04:50:32.905962 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.905768    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 04:50:32.906074 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.906049    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 04:50:32.910946 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.910917    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 04:50:32.916289 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.916265    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tfrst"]
Jul 16 04:50:32.919758 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.919728    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"]
Jul 16 04:50:32.919923 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.919881    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:32.922681 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.922638    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 04:50:32.922681 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.922675    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dcm9d\""
Jul 16 04:50:32.922863 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.922751    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 04:50:32.924201 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.924180    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dk6m5"]
Jul 16 04:50:32.924349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.924319    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:50:32.927481 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.927443    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 16 04:50:32.927642 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.927562    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 16 04:50:32.927713 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.927641    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-5c7tv"]
Jul 16 04:50:32.927713 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.927460    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-j9k26\""
Jul 16 04:50:32.927815 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.927715    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dk6m5"
Jul 16 04:50:32.931884 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.930941    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 04:50:32.931884 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.931219    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2p66f\""
Jul 16 04:50:32.931884 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.931279    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 04:50:32.931884 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.931525    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 04:50:32.935436 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.935409    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-776bb7cd46-j6xbd"]
Jul 16 04:50:32.935436 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.935452    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dk6m5"]
Jul 16 04:50:32.935678 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.935477    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-5c7tv"]
Jul 16 04:50:32.935678 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.935493    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tfrst"]
Jul 16 04:50:32.935678 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.935515    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"]
Jul 16 04:50:32.935678 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.935649    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-5c7tv"
Jul 16 04:50:32.938216 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.938195    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 04:50:32.938701 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.938680    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9l8lp\""
Jul 16 04:50:32.939187 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:32.939153    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 04:50:33.001620 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.001568    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smptp\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-kube-api-access-smptp\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.001787 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.001630    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.001787 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.001663    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-trusted-ca\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.001787 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.001680    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5dedf531-2032-458b-8003-a88d1379a808-nginx-conf\") pod \"networking-console-plugin-6cf856758c-ldqtw\" (UID: \"5dedf531-2032-458b-8003-a88d1379a808\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:50:33.001787 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.001737    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-ca-trust-extracted\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.001787 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.001754    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-certificates\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.001787 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.001770    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4rkn\" (UniqueName: \"kubernetes.io/projected/d2e8927f-33f5-4cce-a369-b7736e189b57-kube-api-access-q4rkn\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:33.001787 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.001785    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgs6w\" (UniqueName: \"kubernetes.io/projected/fa1b46f4-20f2-4f5d-a83e-6be710913cdd-kube-api-access-dgs6w\") pod \"network-check-source-869dcfcdcd-5c7tv\" (UID: \"fa1b46f4-20f2-4f5d-a83e-6be710913cdd\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-5c7tv"
Jul 16 04:50:33.002081 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.001829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:33.002081 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.001872    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-image-registry-private-configuration\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.002183 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.001905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhqlx\" (UniqueName: \"kubernetes.io/projected/31176ca2-45ab-4eb0-abc7-b4ceae534396-kube-api-access-fhqlx\") pod \"ingress-canary-dk6m5\" (UID: \"31176ca2-45ab-4eb0-abc7-b4ceae534396\") " pod="openshift-ingress-canary/ingress-canary-dk6m5"
Jul 16 04:50:33.002323 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.002306    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31176ca2-45ab-4eb0-abc7-b4ceae534396-cert\") pod \"ingress-canary-dk6m5\" (UID: \"31176ca2-45ab-4eb0-abc7-b4ceae534396\") " pod="openshift-ingress-canary/ingress-canary-dk6m5"
Jul 16 04:50:33.002377 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.002350    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ldqtw\" (UID: \"5dedf531-2032-458b-8003-a88d1379a808\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:50:33.002431 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.002399    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e8927f-33f5-4cce-a369-b7736e189b57-config-volume\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:33.002478 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.002430    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e8927f-33f5-4cce-a369-b7736e189b57-tmp-dir\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:33.002584 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.002549    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-installation-pull-secrets\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.002683 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.002628    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-bound-sa-token\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.106421 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106376    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-image-registry-private-configuration\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.106625 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106439    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhqlx\" (UniqueName: \"kubernetes.io/projected/31176ca2-45ab-4eb0-abc7-b4ceae534396-kube-api-access-fhqlx\") pod \"ingress-canary-dk6m5\" (UID: \"31176ca2-45ab-4eb0-abc7-b4ceae534396\") " pod="openshift-ingress-canary/ingress-canary-dk6m5"
Jul 16 04:50:33.106625 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106465    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31176ca2-45ab-4eb0-abc7-b4ceae534396-cert\") pod \"ingress-canary-dk6m5\" (UID: \"31176ca2-45ab-4eb0-abc7-b4ceae534396\") " pod="openshift-ingress-canary/ingress-canary-dk6m5"
Jul 16 04:50:33.106625 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ldqtw\" (UID: \"5dedf531-2032-458b-8003-a88d1379a808\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:50:33.106625 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106525    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e8927f-33f5-4cce-a369-b7736e189b57-config-volume\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:33.106625 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106548    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e8927f-33f5-4cce-a369-b7736e189b57-tmp-dir\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:33.106625 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106579    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-installation-pull-secrets\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.106848 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-bound-sa-token\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.106848 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smptp\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-kube-api-access-smptp\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.106848 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106711    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.106848 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106737    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-trusted-ca\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.106848 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106760    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5dedf531-2032-458b-8003-a88d1379a808-nginx-conf\") pod \"networking-console-plugin-6cf856758c-ldqtw\" (UID: \"5dedf531-2032-458b-8003-a88d1379a808\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:50:33.106848 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-ca-trust-extracted\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.107017 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106852    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-certificates\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.107017 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106874    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4rkn\" (UniqueName: \"kubernetes.io/projected/d2e8927f-33f5-4cce-a369-b7736e189b57-kube-api-access-q4rkn\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:33.107017 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgs6w\" (UniqueName: \"kubernetes.io/projected/fa1b46f4-20f2-4f5d-a83e-6be710913cdd-kube-api-access-dgs6w\") pod \"network-check-source-869dcfcdcd-5c7tv\" (UID: \"fa1b46f4-20f2-4f5d-a83e-6be710913cdd\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-5c7tv"
Jul 16 04:50:33.107017 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.106930    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:33.107180 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:33.107044    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 04:50:33.107180 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:33.107076    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 04:50:33.107180 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:33.107111    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls podName:d2e8927f-33f5-4cce-a369-b7736e189b57 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:33.607086216 +0000 UTC m=+37.023041293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls") pod "dns-default-tfrst" (UID: "d2e8927f-33f5-4cce-a369-b7736e189b57") : secret "dns-default-metrics-tls" not found
Jul 16 04:50:33.107180 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:33.107131    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert podName:5dedf531-2032-458b-8003-a88d1379a808 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:33.607121663 +0000 UTC m=+37.023076715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-ldqtw" (UID: "5dedf531-2032-458b-8003-a88d1379a808") : secret "networking-console-plugin-cert" not found
Jul 16 04:50:33.107539 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.107408    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e8927f-33f5-4cce-a369-b7736e189b57-tmp-dir\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:33.107539 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.107516    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-ca-trust-extracted\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.107720 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:33.107667    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 04:50:33.107720 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:33.107683    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-776bb7cd46-j6xbd: secret "image-registry-tls" not found
Jul 16 04:50:33.108079 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:33.107727    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls podName:40f2c277-95a8-4dc0-acda-1fd724cd9c0f nodeName:}" failed. No retries permitted until 2026-07-16 04:50:33.607711376 +0000 UTC m=+37.023666443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls") pod "image-registry-776bb7cd46-j6xbd" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f") : secret "image-registry-tls" not found
Jul 16 04:50:33.108079 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.107909    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-certificates\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.108079 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.107970    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e8927f-33f5-4cce-a369-b7736e189b57-config-volume\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:33.108263 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.108153    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5dedf531-2032-458b-8003-a88d1379a808-nginx-conf\") pod \"networking-console-plugin-6cf856758c-ldqtw\" (UID: \"5dedf531-2032-458b-8003-a88d1379a808\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:50:33.108339 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.108314    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-trusted-ca\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.112110 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.112078    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-image-registry-private-configuration\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.112249 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.112109    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-installation-pull-secrets\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.112249 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.112137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31176ca2-45ab-4eb0-abc7-b4ceae534396-cert\") pod \"ingress-canary-dk6m5\" (UID: \"31176ca2-45ab-4eb0-abc7-b4ceae534396\") " pod="openshift-ingress-canary/ingress-canary-dk6m5"
Jul 16 04:50:33.122146 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.122112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhqlx\" (UniqueName: \"kubernetes.io/projected/31176ca2-45ab-4eb0-abc7-b4ceae534396-kube-api-access-fhqlx\") pod \"ingress-canary-dk6m5\" (UID: \"31176ca2-45ab-4eb0-abc7-b4ceae534396\") " pod="openshift-ingress-canary/ingress-canary-dk6m5"
Jul 16 04:50:33.122298 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.122168    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-bound-sa-token\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.122298 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.122240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smptp\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-kube-api-access-smptp\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.122298 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.122280    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgs6w\" (UniqueName: \"kubernetes.io/projected/fa1b46f4-20f2-4f5d-a83e-6be710913cdd-kube-api-access-dgs6w\") pod \"network-check-source-869dcfcdcd-5c7tv\" (UID: \"fa1b46f4-20f2-4f5d-a83e-6be710913cdd\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-5c7tv"
Jul 16 04:50:33.122455 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.122353    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4rkn\" (UniqueName: \"kubernetes.io/projected/d2e8927f-33f5-4cce-a369-b7736e189b57-kube-api-access-q4rkn\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:33.136510 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.136470    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:50:33.136510 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.136504    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:50:33.136767 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.136470    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:50:33.139767 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.139476    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vpdjw\""
Jul 16 04:50:33.139767 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.139488    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 04:50:33.139767 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.139510    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 04:50:33.139767 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.139556    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-njf2v\""
Jul 16 04:50:33.249905 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.249876    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dk6m5"
Jul 16 04:50:33.255056 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.254862    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-5c7tv"
Jul 16 04:50:33.611885 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.611848    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:33.612081 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.611917    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:33.612081 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:33.612023    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 04:50:33.612081 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:33.612028    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 04:50:33.612081 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:33.612049    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-776bb7cd46-j6xbd: secret "image-registry-tls" not found
Jul 16 04:50:33.612299 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:33.612096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ldqtw\" (UID: \"5dedf531-2032-458b-8003-a88d1379a808\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:50:33.612299 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:33.612119    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls podName:40f2c277-95a8-4dc0-acda-1fd724cd9c0f nodeName:}" failed. No retries permitted until 2026-07-16 04:50:34.612096981 +0000 UTC m=+38.028052055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls") pod "image-registry-776bb7cd46-j6xbd" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f") : secret "image-registry-tls" not found
Jul 16 04:50:33.612299 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:33.612138    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls podName:d2e8927f-33f5-4cce-a369-b7736e189b57 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:34.612129782 +0000 UTC m=+38.028084833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls") pod "dns-default-tfrst" (UID: "d2e8927f-33f5-4cce-a369-b7736e189b57") : secret "dns-default-metrics-tls" not found
Jul 16 04:50:33.612299 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:33.612191    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 04:50:33.612299 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:33.612256    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert podName:5dedf531-2032-458b-8003-a88d1379a808 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:34.612240176 +0000 UTC m=+38.028195240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-ldqtw" (UID: "5dedf531-2032-458b-8003-a88d1379a808") : secret "networking-console-plugin-cert" not found
Jul 16 04:50:34.531541 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:34.531498    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dk6m5"]
Jul 16 04:50:34.532457 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:34.532430    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-5c7tv"]
Jul 16 04:50:34.535888 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:50:34.535860    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1b46f4_20f2_4f5d_a83e_6be710913cdd.slice/crio-a068c382220b7388d08dde5d9b2c456f9d40e0ef4c4c003de8c11cd99740e65c WatchSource:0}: Error finding container a068c382220b7388d08dde5d9b2c456f9d40e0ef4c4c003de8c11cd99740e65c: Status 404 returned error can't find the container with id a068c382220b7388d08dde5d9b2c456f9d40e0ef4c4c003de8c11cd99740e65c
Jul 16 04:50:34.541745 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:50:34.541718    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31176ca2_45ab_4eb0_abc7_b4ceae534396.slice/crio-eeff32470d2f2d24d9dc6a090a5ce5f5c4795cbac4b2a75a33c5604b5e3f06e8 WatchSource:0}: Error finding container eeff32470d2f2d24d9dc6a090a5ce5f5c4795cbac4b2a75a33c5604b5e3f06e8: Status 404 returned error can't find the container with id eeff32470d2f2d24d9dc6a090a5ce5f5c4795cbac4b2a75a33c5604b5e3f06e8
Jul 16 04:50:34.620878 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:34.620847    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:34.621007 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:34.620889    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ldqtw\" (UID: \"5dedf531-2032-458b-8003-a88d1379a808\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:50:34.621007 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:34.620937    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:34.621085 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:34.621029    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 04:50:34.621119 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:34.621097    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert podName:5dedf531-2032-458b-8003-a88d1379a808 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:36.621081921 +0000 UTC m=+40.037036976 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-ldqtw" (UID: "5dedf531-2032-458b-8003-a88d1379a808") : secret "networking-console-plugin-cert" not found
Jul 16 04:50:34.621170 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:34.621037    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 04:50:34.621170 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:34.621040    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 04:50:34.621170 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:34.621163    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls podName:d2e8927f-33f5-4cce-a369-b7736e189b57 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:36.621149455 +0000 UTC m=+40.037104506 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls") pod "dns-default-tfrst" (UID: "d2e8927f-33f5-4cce-a369-b7736e189b57") : secret "dns-default-metrics-tls" not found
Jul 16 04:50:34.621170 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:34.621166    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-776bb7cd46-j6xbd: secret "image-registry-tls" not found
Jul 16 04:50:34.621298 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:34.621222    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls podName:40f2c277-95a8-4dc0-acda-1fd724cd9c0f nodeName:}" failed. No retries permitted until 2026-07-16 04:50:36.621204983 +0000 UTC m=+40.037160056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls") pod "image-registry-776bb7cd46-j6xbd" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f") : secret "image-registry-tls" not found
Jul 16 04:50:35.365807 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:35.365544    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-5c7tv" event={"ID":"fa1b46f4-20f2-4f5d-a83e-6be710913cdd","Type":"ContainerStarted","Data":"a068c382220b7388d08dde5d9b2c456f9d40e0ef4c4c003de8c11cd99740e65c"}
Jul 16 04:50:35.367128 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:35.367089    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dk6m5" event={"ID":"31176ca2-45ab-4eb0-abc7-b4ceae534396","Type":"ContainerStarted","Data":"eeff32470d2f2d24d9dc6a090a5ce5f5c4795cbac4b2a75a33c5604b5e3f06e8"}
Jul 16 04:50:35.370994 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:35.370962    2565 generic.go:356] "Generic (PLEG): container finished" podID="ec34c160-0104-4e3e-9e81-bd84e8266708" containerID="685e1ee3cc800ff3980dcddb747f53a664a58850f771872d3fc78eed35121d0c" exitCode=0
Jul 16 04:50:35.371116 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:35.371027    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fsdp" event={"ID":"ec34c160-0104-4e3e-9e81-bd84e8266708","Type":"ContainerDied","Data":"685e1ee3cc800ff3980dcddb747f53a664a58850f771872d3fc78eed35121d0c"}
Jul 16 04:50:36.376793 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:36.376755    2565 generic.go:356] "Generic (PLEG): container finished" podID="ec34c160-0104-4e3e-9e81-bd84e8266708" containerID="063bb93c082837fd297596914a03a099590de242a38d0184ed1878b59cdcf2c5" exitCode=0
Jul 16 04:50:36.376793 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:36.376800    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fsdp" event={"ID":"ec34c160-0104-4e3e-9e81-bd84e8266708","Type":"ContainerDied","Data":"063bb93c082837fd297596914a03a099590de242a38d0184ed1878b59cdcf2c5"}
Jul 16 04:50:36.639431 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:36.639341    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:36.639431 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:36.639415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:36.639672 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:36.639452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ldqtw\" (UID: \"5dedf531-2032-458b-8003-a88d1379a808\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:50:36.639672 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:36.639613    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 04:50:36.639783 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:36.639682    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert podName:5dedf531-2032-458b-8003-a88d1379a808 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:40.639662274 +0000 UTC m=+44.055617343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-ldqtw" (UID: "5dedf531-2032-458b-8003-a88d1379a808") : secret "networking-console-plugin-cert" not found
Jul 16 04:50:36.639783 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:36.639689    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 04:50:36.639783 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:36.639703    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-776bb7cd46-j6xbd: secret "image-registry-tls" not found
Jul 16 04:50:36.639783 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:36.639725    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 04:50:36.639783 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:36.639736    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls podName:40f2c277-95a8-4dc0-acda-1fd724cd9c0f nodeName:}" failed. No retries permitted until 2026-07-16 04:50:40.639725956 +0000 UTC m=+44.055681023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls") pod "image-registry-776bb7cd46-j6xbd" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f") : secret "image-registry-tls" not found
Jul 16 04:50:36.640043 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:36.639804    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls podName:d2e8927f-33f5-4cce-a369-b7736e189b57 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:40.639783988 +0000 UTC m=+44.055739040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls") pod "dns-default-tfrst" (UID: "d2e8927f-33f5-4cce-a369-b7736e189b57") : secret "dns-default-metrics-tls" not found
Jul 16 04:50:39.387665 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:39.387628    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fsdp" event={"ID":"ec34c160-0104-4e3e-9e81-bd84e8266708","Type":"ContainerStarted","Data":"ab540f5fb9549011fa0861d1b0f9bc864562d6c6ddbea01389aa7e1ff0e227ec"}
Jul 16 04:50:39.389208 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:39.389164    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-5c7tv" event={"ID":"fa1b46f4-20f2-4f5d-a83e-6be710913cdd","Type":"ContainerStarted","Data":"f725998b3736f40e31f858386b0b13939b56b81484529a1ba233b4e1bc58335c"}
Jul 16 04:50:39.390725 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:39.390692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dk6m5" event={"ID":"31176ca2-45ab-4eb0-abc7-b4ceae534396","Type":"ContainerStarted","Data":"eee4a9ab31b4026b49619cd78449c9deeae65ffde053466af29c94299ff5b038"}
Jul 16 04:50:39.409462 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:39.409362    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9fsdp" podStartSLOduration=10.290156071 podStartE2EDuration="41.409337938s" totalImagesPullingTime="31.119181867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:49:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:50:39.407481281 +0000 UTC m=+42.823436353" watchObservedRunningTime="2026-07-16 04:50:39.409337938 +0000 UTC m=+42.825293012"
Jul 16 04:50:39.424829 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:39.424757    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dk6m5" podStartSLOduration=3.5722041129999997 podStartE2EDuration="7.424733353s" totalImagesPullingTime="3.85252924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:50:39.423283732 +0000 UTC m=+42.839238808" watchObservedRunningTime="2026-07-16 04:50:39.424733353 +0000 UTC m=+42.840688430"
Jul 16 04:50:39.438213 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:39.438153    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-5c7tv" podStartSLOduration=33.584686481 podStartE2EDuration="37.438132087s" totalImagesPullingTime="3.853445606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:50:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:50:39.437363221 +0000 UTC m=+42.853318294" watchObservedRunningTime="2026-07-16 04:50:39.438132087 +0000 UTC m=+42.854087160"
Jul 16 04:50:40.672692 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:40.672642    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:40.673139 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:40.672729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:40.673139 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:40.672762    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ldqtw\" (UID: \"5dedf531-2032-458b-8003-a88d1379a808\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:50:40.673139 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:40.672794    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 04:50:40.673139 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:40.672815    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-776bb7cd46-j6xbd: secret "image-registry-tls" not found
Jul 16 04:50:40.673139 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:40.672859    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 04:50:40.673139 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:40.672881    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls podName:40f2c277-95a8-4dc0-acda-1fd724cd9c0f nodeName:}" failed. No retries permitted until 2026-07-16 04:50:48.672863555 +0000 UTC m=+52.088818606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls") pod "image-registry-776bb7cd46-j6xbd" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f") : secret "image-registry-tls" not found
Jul 16 04:50:40.673139 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:40.672908    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 04:50:40.673139 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:40.672915    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls podName:d2e8927f-33f5-4cce-a369-b7736e189b57 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:48.672903794 +0000 UTC m=+52.088858849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls") pod "dns-default-tfrst" (UID: "d2e8927f-33f5-4cce-a369-b7736e189b57") : secret "dns-default-metrics-tls" not found
Jul 16 04:50:40.673139 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:40.672971    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert podName:5dedf531-2032-458b-8003-a88d1379a808 nodeName:}" failed. No retries permitted until 2026-07-16 04:50:48.672958576 +0000 UTC m=+52.088913663 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-ldqtw" (UID: "5dedf531-2032-458b-8003-a88d1379a808") : secret "networking-console-plugin-cert" not found
Jul 16 04:50:48.731179 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:48.731134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:50:48.731179 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:48.731179    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ldqtw\" (UID: \"5dedf531-2032-458b-8003-a88d1379a808\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:50:48.731716 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:48.731225    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:50:48.731716 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:48.731300    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 04:50:48.731716 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:48.731334    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 04:50:48.731716 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:48.731346    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-776bb7cd46-j6xbd: secret "image-registry-tls" not found
Jul 16 04:50:48.731716 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:48.731348    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 04:50:48.731716 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:48.731371    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls podName:d2e8927f-33f5-4cce-a369-b7736e189b57 nodeName:}" failed. No retries permitted until 2026-07-16 04:51:04.731353939 +0000 UTC m=+68.147308994 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls") pod "dns-default-tfrst" (UID: "d2e8927f-33f5-4cce-a369-b7736e189b57") : secret "dns-default-metrics-tls" not found
Jul 16 04:50:48.731716 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:48.731389    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls podName:40f2c277-95a8-4dc0-acda-1fd724cd9c0f nodeName:}" failed. No retries permitted until 2026-07-16 04:51:04.731378132 +0000 UTC m=+68.147333183 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls") pod "image-registry-776bb7cd46-j6xbd" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f") : secret "image-registry-tls" not found
Jul 16 04:50:48.731716 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:50:48.731402    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert podName:5dedf531-2032-458b-8003-a88d1379a808 nodeName:}" failed. No retries permitted until 2026-07-16 04:51:04.731396624 +0000 UTC m=+68.147351675 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-ldqtw" (UID: "5dedf531-2032-458b-8003-a88d1379a808") : secret "networking-console-plugin-cert" not found
Jul 16 04:50:59.355660 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:50:59.355619    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5nsc5"
Jul 16 04:51:02.837970 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:02.837915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6fr\" (UniqueName: \"kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr\") pod \"network-check-target-pz5t7\" (UID: \"b74a6491-0f27-40ee-8f69-b0480a1d1889\") " pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:51:02.841775 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:02.841740    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql6fr\" (UniqueName: \"kubernetes.io/projected/b74a6491-0f27-40ee-8f69-b0480a1d1889-kube-api-access-ql6fr\") pod \"network-check-target-pz5t7\" (UID: \"b74a6491-0f27-40ee-8f69-b0480a1d1889\") " pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:51:02.859458 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:02.859425    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-njf2v\""
Jul 16 04:51:02.867538 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:02.867508    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:51:02.938784 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:02.938756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret\") pod \"global-pull-secret-syncer-dmm65\" (UID: \"74612e83-5ffe-4c5d-830b-8e3f83fd8579\") " pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:51:02.938935 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:02.938790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs\") pod \"network-metrics-daemon-ldxlj\" (UID: \"c028e133-a290-4758-998c-24e158389825\") " pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:51:02.941747 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:02.941720    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 04:51:02.941868 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:02.941765    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 04:51:02.949178 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:51:02.949153    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 16 04:51:02.949245 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:51:02.949237    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs podName:c028e133-a290-4758-998c-24e158389825 nodeName:}" failed. No retries permitted until 2026-07-16 04:52:06.949211131 +0000 UTC m=+130.365166186 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs") pod "network-metrics-daemon-ldxlj" (UID: "c028e133-a290-4758-998c-24e158389825") : secret "metrics-daemon-secret" not found
Jul 16 04:51:02.951876 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:02.951816    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74612e83-5ffe-4c5d-830b-8e3f83fd8579-original-pull-secret\") pod \"global-pull-secret-syncer-dmm65\" (UID: \"74612e83-5ffe-4c5d-830b-8e3f83fd8579\") " pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:51:03.003925 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:03.003892    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pz5t7"]
Jul 16 04:51:03.009326 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:51:03.009284    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74a6491_0f27_40ee_8f69_b0480a1d1889.slice/crio-a32af3d8fbe9109f858db367c4ba86e97434270d3e0a0eebf9d41c6ceb284355 WatchSource:0}: Error finding container a32af3d8fbe9109f858db367c4ba86e97434270d3e0a0eebf9d41c6ceb284355: Status 404 returned error can't find the container with id a32af3d8fbe9109f858db367c4ba86e97434270d3e0a0eebf9d41c6ceb284355
Jul 16 04:51:03.148936 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:03.148858    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmm65"
Jul 16 04:51:03.270212 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:03.270180    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dmm65"]
Jul 16 04:51:03.273573 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:51:03.273549    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74612e83_5ffe_4c5d_830b_8e3f83fd8579.slice/crio-ac568b75b94bd0d8dbce7ee0c5d1dfa0d7d872f1005a8c9a4c2751d119ad2a62 WatchSource:0}: Error finding container ac568b75b94bd0d8dbce7ee0c5d1dfa0d7d872f1005a8c9a4c2751d119ad2a62: Status 404 returned error can't find the container with id ac568b75b94bd0d8dbce7ee0c5d1dfa0d7d872f1005a8c9a4c2751d119ad2a62
Jul 16 04:51:03.448941 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:03.448859    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dmm65" event={"ID":"74612e83-5ffe-4c5d-830b-8e3f83fd8579","Type":"ContainerStarted","Data":"ac568b75b94bd0d8dbce7ee0c5d1dfa0d7d872f1005a8c9a4c2751d119ad2a62"}
Jul 16 04:51:03.450282 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:03.450255    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pz5t7" event={"ID":"b74a6491-0f27-40ee-8f69-b0480a1d1889","Type":"ContainerStarted","Data":"3d9960701cee3e573a55549b8c396e43d5beb21e898d9bd03d10b296f32598b7"}
Jul 16 04:51:03.450408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:03.450304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pz5t7" event={"ID":"b74a6491-0f27-40ee-8f69-b0480a1d1889","Type":"ContainerStarted","Data":"a32af3d8fbe9109f858db367c4ba86e97434270d3e0a0eebf9d41c6ceb284355"}
Jul 16 04:51:03.450408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:03.450398    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:51:03.465637 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:03.465567    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pz5t7" podStartSLOduration=65.465551108 podStartE2EDuration="1m5.465551108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:51:03.465108664 +0000 UTC m=+66.881063737" watchObservedRunningTime="2026-07-16 04:51:03.465551108 +0000 UTC m=+66.881506175"
Jul 16 04:51:04.750303 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:04.750259    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:51:04.750848 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:04.750316    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:51:04.750848 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:51:04.750407    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 04:51:04.750848 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:51:04.750412    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 04:51:04.750848 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:51:04.750434    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-776bb7cd46-j6xbd: secret "image-registry-tls" not found
Jul 16 04:51:04.750848 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:51:04.750452    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls podName:d2e8927f-33f5-4cce-a369-b7736e189b57 nodeName:}" failed. No retries permitted until 2026-07-16 04:51:36.750439527 +0000 UTC m=+100.166394594 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls") pod "dns-default-tfrst" (UID: "d2e8927f-33f5-4cce-a369-b7736e189b57") : secret "dns-default-metrics-tls" not found
Jul 16 04:51:04.750848 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:04.750454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ldqtw\" (UID: \"5dedf531-2032-458b-8003-a88d1379a808\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:51:04.750848 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:51:04.750474    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls podName:40f2c277-95a8-4dc0-acda-1fd724cd9c0f nodeName:}" failed. No retries permitted until 2026-07-16 04:51:36.750462159 +0000 UTC m=+100.166417210 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls") pod "image-registry-776bb7cd46-j6xbd" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f") : secret "image-registry-tls" not found
Jul 16 04:51:04.750848 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:51:04.750519    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 04:51:04.750848 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:51:04.750548    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert podName:5dedf531-2032-458b-8003-a88d1379a808 nodeName:}" failed. No retries permitted until 2026-07-16 04:51:36.750540958 +0000 UTC m=+100.166496008 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-ldqtw" (UID: "5dedf531-2032-458b-8003-a88d1379a808") : secret "networking-console-plugin-cert" not found
Jul 16 04:51:08.465068 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:08.465023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dmm65" event={"ID":"74612e83-5ffe-4c5d-830b-8e3f83fd8579","Type":"ContainerStarted","Data":"7d4bee5c74257e32197e7783ade4cb7b3b871a779ff5dc5db2db15ac9c3daa01"}
Jul 16 04:51:08.479474 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:08.479423    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dmm65" podStartSLOduration=66.133224223 podStartE2EDuration="1m10.479406068s" totalImagesPullingTime="4.346181845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:51:08.478681918 +0000 UTC m=+71.894637004" watchObservedRunningTime="2026-07-16 04:51:08.479406068 +0000 UTC m=+71.895361140"
Jul 16 04:51:34.456393 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:34.456359    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pz5t7"
Jul 16 04:51:36.793174 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:36.793123    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:51:36.793174 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:36.793180    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ldqtw\" (UID: \"5dedf531-2032-458b-8003-a88d1379a808\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:51:36.793665 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:51:36.793241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:51:36.793665 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:51:36.793304    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 04:51:36.793665 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:51:36.793336    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 04:51:36.793665 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:51:36.793348    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-776bb7cd46-j6xbd: secret "image-registry-tls" not found
Jul 16 04:51:36.793665 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:51:36.793348    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 04:51:36.793665 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:51:36.793385    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls podName:d2e8927f-33f5-4cce-a369-b7736e189b57 nodeName:}" failed. No retries permitted until 2026-07-16 04:52:40.793362155 +0000 UTC m=+164.209317209 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls") pod "dns-default-tfrst" (UID: "d2e8927f-33f5-4cce-a369-b7736e189b57") : secret "dns-default-metrics-tls" not found
Jul 16 04:51:36.793665 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:51:36.793403    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls podName:40f2c277-95a8-4dc0-acda-1fd724cd9c0f nodeName:}" failed. No retries permitted until 2026-07-16 04:52:40.793395259 +0000 UTC m=+164.209350315 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls") pod "image-registry-776bb7cd46-j6xbd" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f") : secret "image-registry-tls" not found
Jul 16 04:51:36.793665 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:51:36.793414    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert podName:5dedf531-2032-458b-8003-a88d1379a808 nodeName:}" failed. No retries permitted until 2026-07-16 04:52:40.793408499 +0000 UTC m=+164.209363549 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-ldqtw" (UID: "5dedf531-2032-458b-8003-a88d1379a808") : secret "networking-console-plugin-cert" not found
Jul 16 04:52:07.017250 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:52:07.017193    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs\") pod \"network-metrics-daemon-ldxlj\" (UID: \"c028e133-a290-4758-998c-24e158389825\") " pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:52:07.017782 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:52:07.017348    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 16 04:52:07.017782 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:52:07.017418    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs podName:c028e133-a290-4758-998c-24e158389825 nodeName:}" failed. No retries permitted until 2026-07-16 04:54:09.017400665 +0000 UTC m=+252.433355716 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs") pod "network-metrics-daemon-ldxlj" (UID: "c028e133-a290-4758-998c-24e158389825") : secret "metrics-daemon-secret" not found
Jul 16 04:52:35.915716 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:52:35.915653    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd" podUID="40f2c277-95a8-4dc0-acda-1fd724cd9c0f"
Jul 16 04:52:35.933945 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:52:35.933919    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tfrst" podUID="d2e8927f-33f5-4cce-a369-b7736e189b57"
Jul 16 04:52:35.942283 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:52:35.942248    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw" podUID="5dedf531-2032-458b-8003-a88d1379a808"
Jul 16 04:52:36.162663 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:52:36.162615    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:52:36.679857 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:52:36.679822    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:52:36.679857 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:52:36.679845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tfrst"
Jul 16 04:52:36.679857 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:52:36.679837    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:52:40.859763 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:52:40.859726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:52:40.859763 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:52:40.859765    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ldqtw\" (UID: \"5dedf531-2032-458b-8003-a88d1379a808\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:52:40.860260 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:52:40.859807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:52:40.860260 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:52:40.859868    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 04:52:40.860260 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:52:40.859891    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 04:52:40.860260 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:52:40.859905    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-776bb7cd46-j6xbd: secret "image-registry-tls" not found
Jul 16 04:52:40.860260 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:52:40.859945    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls podName:d2e8927f-33f5-4cce-a369-b7736e189b57 nodeName:}" failed. No retries permitted until 2026-07-16 04:54:42.859929087 +0000 UTC m=+286.275884139 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls") pod "dns-default-tfrst" (UID: "d2e8927f-33f5-4cce-a369-b7736e189b57") : secret "dns-default-metrics-tls" not found
Jul 16 04:52:40.860260 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:52:40.859954    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 04:52:40.860260 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:52:40.859959    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls podName:40f2c277-95a8-4dc0-acda-1fd724cd9c0f nodeName:}" failed. No retries permitted until 2026-07-16 04:54:42.859953667 +0000 UTC m=+286.275908718 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls") pod "image-registry-776bb7cd46-j6xbd" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f") : secret "image-registry-tls" not found
Jul 16 04:52:40.860260 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:52:40.860040    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert podName:5dedf531-2032-458b-8003-a88d1379a808 nodeName:}" failed. No retries permitted until 2026-07-16 04:54:42.860021432 +0000 UTC m=+286.275976483 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-ldqtw" (UID: "5dedf531-2032-458b-8003-a88d1379a808") : secret "networking-console-plugin-cert" not found
Jul 16 04:52:47.139551 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:52:47.139513    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:54:09.075162 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:09.075095    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs\") pod \"network-metrics-daemon-ldxlj\" (UID: \"c028e133-a290-4758-998c-24e158389825\") " pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:54:09.075721 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:09.075268    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 16 04:54:09.075721 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:09.075335    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs podName:c028e133-a290-4758-998c-24e158389825 nodeName:}" failed. No retries permitted until 2026-07-16 04:56:11.075317491 +0000 UTC m=+374.491272543 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs") pod "network-metrics-daemon-ldxlj" (UID: "c028e133-a290-4758-998c-24e158389825") : secret "metrics-daemon-secret" not found
Jul 16 04:54:18.265384 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.265292    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs"]
Jul 16 04:54:18.268268 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.268243    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"]
Jul 16 04:54:18.268424 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.268362    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs"
Jul 16 04:54:18.271104 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.271079    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7"]
Jul 16 04:54:18.271234 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.271201    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"
Jul 16 04:54:18.274135 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.274109    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-msc9n"]
Jul 16 04:54:18.274261 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.274234    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7"
Jul 16 04:54:18.276425 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.276404    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 16 04:54:18.276961 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.276934    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:18.279929 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.279908    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 16 04:54:18.280847 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.280828    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 16 04:54:18.284292 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.284256    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 04:54:18.284292 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.284288    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-l7hwf\""
Jul 16 04:54:18.284510 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.284290    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 16 04:54:18.284510 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.284256    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 16 04:54:18.284790 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.284774    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 16 04:54:18.284878 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.284810    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 16 04:54:18.284878 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.284846    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 16 04:54:18.284992 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.284850    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 16 04:54:18.284992 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.284981    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 16 04:54:18.285090 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.285071    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 16 04:54:18.285139 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.285088    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 16 04:54:18.285185 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.285153    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9ht8t\""
Jul 16 04:54:18.285233 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.285194    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 16 04:54:18.285283 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.285194    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 04:54:18.286661 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.286641    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p9zzx\""
Jul 16 04:54:18.287695 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.287672    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8qmmv\""
Jul 16 04:54:18.295225 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.294691    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs"]
Jul 16 04:54:18.299013 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.298982    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"]
Jul 16 04:54:18.299761 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.299742    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7"]
Jul 16 04:54:18.305630 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.305580    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-msc9n"]
Jul 16 04:54:18.307455 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.307396    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 16 04:54:18.344499 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.344444    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9582f0df-040b-4ff8-8eca-53207122af5b-config\") pod \"console-operator-7866497494-msc9n\" (UID: \"9582f0df-040b-4ff8-8eca-53207122af5b\") " pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:18.344499 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.344507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwwvr\" (UniqueName: \"kubernetes.io/projected/22b39ae9-d2af-4874-9225-bd9a7a7f6bbf-kube-api-access-kwwvr\") pod \"service-ca-operator-7695fc6c8c-tcnzs\" (UID: \"22b39ae9-d2af-4874-9225-bd9a7a7f6bbf\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs"
Jul 16 04:54:18.344777 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.344555    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ba263a6a-62d7-412e-9459-04f53586cdbb-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-k6lpx\" (UID: \"ba263a6a-62d7-412e-9459-04f53586cdbb\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"
Jul 16 04:54:18.344777 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.344581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9582f0df-040b-4ff8-8eca-53207122af5b-serving-cert\") pod \"console-operator-7866497494-msc9n\" (UID: \"9582f0df-040b-4ff8-8eca-53207122af5b\") " pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:18.344777 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.344672    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/22b39ae9-d2af-4874-9225-bd9a7a7f6bbf-config\") pod \"service-ca-operator-7695fc6c8c-tcnzs\" (UID: \"22b39ae9-d2af-4874-9225-bd9a7a7f6bbf\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs"
Jul 16 04:54:18.344777 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.344703    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc7ps\" (UniqueName: \"kubernetes.io/projected/9582f0df-040b-4ff8-8eca-53207122af5b-kube-api-access-hc7ps\") pod \"console-operator-7866497494-msc9n\" (UID: \"9582f0df-040b-4ff8-8eca-53207122af5b\") " pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:18.344777 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.344749    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-k6lpx\" (UID: \"ba263a6a-62d7-412e-9459-04f53586cdbb\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"
Jul 16 04:54:18.344987 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.344779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-cz8w7\" (UID: \"22f69360-ae6d-446f-9267-daba8e58ed2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7"
Jul 16 04:54:18.344987 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.344809    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqnq6\" (UniqueName: \"kubernetes.io/projected/ba263a6a-62d7-412e-9459-04f53586cdbb-kube-api-access-vqnq6\") pod \"cluster-monitoring-operator-5f7f494dd4-k6lpx\" (UID: \"ba263a6a-62d7-412e-9459-04f53586cdbb\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"
Jul 16 04:54:18.344987 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.344826    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q56n\" (UniqueName: \"kubernetes.io/projected/22f69360-ae6d-446f-9267-daba8e58ed2b-kube-api-access-4q56n\") pod \"cluster-samples-operator-65db858894-cz8w7\" (UID: \"22f69360-ae6d-446f-9267-daba8e58ed2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7"
Jul 16 04:54:18.344987 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.344862    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22b39ae9-d2af-4874-9225-bd9a7a7f6bbf-serving-cert\") pod \"service-ca-operator-7695fc6c8c-tcnzs\" (UID: \"22b39ae9-d2af-4874-9225-bd9a7a7f6bbf\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs"
Jul 16 04:54:18.344987 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.344893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9582f0df-040b-4ff8-8eca-53207122af5b-trusted-ca\") pod \"console-operator-7866497494-msc9n\" (UID: \"9582f0df-040b-4ff8-8eca-53207122af5b\") " pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:18.360652 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.360618    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8"]
Jul 16 04:54:18.363756 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.363734    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8"
Jul 16 04:54:18.366416 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.366381    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 16 04:54:18.366416 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.366407    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 16 04:54:18.366763 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.366747    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 16 04:54:18.366825 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.366775    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 16 04:54:18.366825 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.366806    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9vklh\""
Jul 16 04:54:18.375609 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.375556    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8"]
Jul 16 04:54:18.445501 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.445456    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqnq6\" (UniqueName: \"kubernetes.io/projected/ba263a6a-62d7-412e-9459-04f53586cdbb-kube-api-access-vqnq6\") pod \"cluster-monitoring-operator-5f7f494dd4-k6lpx\" (UID: \"ba263a6a-62d7-412e-9459-04f53586cdbb\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"
Jul 16 04:54:18.445501 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.445503    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q56n\" (UniqueName: \"kubernetes.io/projected/22f69360-ae6d-446f-9267-daba8e58ed2b-kube-api-access-4q56n\") pod \"cluster-samples-operator-65db858894-cz8w7\" (UID: \"22f69360-ae6d-446f-9267-daba8e58ed2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7"
Jul 16 04:54:18.445791 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.445531    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22b39ae9-d2af-4874-9225-bd9a7a7f6bbf-serving-cert\") pod \"service-ca-operator-7695fc6c8c-tcnzs\" (UID: \"22b39ae9-d2af-4874-9225-bd9a7a7f6bbf\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs"
Jul 16 04:54:18.445791 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.445547    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9582f0df-040b-4ff8-8eca-53207122af5b-trusted-ca\") pod \"console-operator-7866497494-msc9n\" (UID: \"9582f0df-040b-4ff8-8eca-53207122af5b\") " pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:18.445791 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.445573    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9582f0df-040b-4ff8-8eca-53207122af5b-config\") pod \"console-operator-7866497494-msc9n\" (UID: \"9582f0df-040b-4ff8-8eca-53207122af5b\") " pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:18.445791 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.445622    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggz77\" (UniqueName: \"kubernetes.io/projected/95bf5f5b-d8bb-4638-895a-015e35aea09a-kube-api-access-ggz77\") pod \"kube-storage-version-migrator-operator-799cb7f958-2xgh8\" (UID: \"95bf5f5b-d8bb-4638-895a-015e35aea09a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8"
Jul 16 04:54:18.445791 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.445647    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwwvr\" (UniqueName: \"kubernetes.io/projected/22b39ae9-d2af-4874-9225-bd9a7a7f6bbf-kube-api-access-kwwvr\") pod \"service-ca-operator-7695fc6c8c-tcnzs\" (UID: \"22b39ae9-d2af-4874-9225-bd9a7a7f6bbf\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs"
Jul 16 04:54:18.445791 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.445688    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ba263a6a-62d7-412e-9459-04f53586cdbb-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-k6lpx\" (UID: \"ba263a6a-62d7-412e-9459-04f53586cdbb\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"
Jul 16 04:54:18.445791 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.445715    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95bf5f5b-d8bb-4638-895a-015e35aea09a-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-2xgh8\" (UID: \"95bf5f5b-d8bb-4638-895a-015e35aea09a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8"
Jul 16 04:54:18.445791 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.445746    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9582f0df-040b-4ff8-8eca-53207122af5b-serving-cert\") pod \"console-operator-7866497494-msc9n\" (UID: \"9582f0df-040b-4ff8-8eca-53207122af5b\") " pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:18.445791 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.445766    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95bf5f5b-d8bb-4638-895a-015e35aea09a-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-2xgh8\" (UID: \"95bf5f5b-d8bb-4638-895a-015e35aea09a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8"
Jul 16 04:54:18.445791 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.445790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/22b39ae9-d2af-4874-9225-bd9a7a7f6bbf-config\") pod \"service-ca-operator-7695fc6c8c-tcnzs\" (UID: \"22b39ae9-d2af-4874-9225-bd9a7a7f6bbf\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs"
Jul 16 04:54:18.446269 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.445875    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc7ps\" (UniqueName: \"kubernetes.io/projected/9582f0df-040b-4ff8-8eca-53207122af5b-kube-api-access-hc7ps\") pod \"console-operator-7866497494-msc9n\" (UID: \"9582f0df-040b-4ff8-8eca-53207122af5b\") " pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:18.446269 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.445938    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-k6lpx\" (UID: \"ba263a6a-62d7-412e-9459-04f53586cdbb\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"
Jul 16 04:54:18.446269 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.445982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-cz8w7\" (UID: \"22f69360-ae6d-446f-9267-daba8e58ed2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7"
Jul 16 04:54:18.446269 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:18.446240    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 04:54:18.446269 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:18.446254    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 04:54:18.446521 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:18.446346    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls podName:22f69360-ae6d-446f-9267-daba8e58ed2b nodeName:}" failed. No retries permitted until 2026-07-16 04:54:18.946289494 +0000 UTC m=+262.362244550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls") pod "cluster-samples-operator-65db858894-cz8w7" (UID: "22f69360-ae6d-446f-9267-daba8e58ed2b") : secret "samples-operator-tls" not found
Jul 16 04:54:18.446521 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:18.446366    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls podName:ba263a6a-62d7-412e-9459-04f53586cdbb nodeName:}" failed. No retries permitted until 2026-07-16 04:54:18.946356214 +0000 UTC m=+262.362311265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-k6lpx" (UID: "ba263a6a-62d7-412e-9459-04f53586cdbb") : secret "cluster-monitoring-operator-tls" not found
Jul 16 04:54:18.446662 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.446613    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9582f0df-040b-4ff8-8eca-53207122af5b-config\") pod \"console-operator-7866497494-msc9n\" (UID: \"9582f0df-040b-4ff8-8eca-53207122af5b\") " pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:18.446722 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.446658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ba263a6a-62d7-412e-9459-04f53586cdbb-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-k6lpx\" (UID: \"ba263a6a-62d7-412e-9459-04f53586cdbb\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"
Jul 16 04:54:18.447001 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.446979    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9582f0df-040b-4ff8-8eca-53207122af5b-trusted-ca\") pod \"console-operator-7866497494-msc9n\" (UID: \"9582f0df-040b-4ff8-8eca-53207122af5b\") " pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:18.447084 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.447063    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/22b39ae9-d2af-4874-9225-bd9a7a7f6bbf-config\") pod \"service-ca-operator-7695fc6c8c-tcnzs\" (UID: \"22b39ae9-d2af-4874-9225-bd9a7a7f6bbf\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs"
Jul 16 04:54:18.448285 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.448256    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22b39ae9-d2af-4874-9225-bd9a7a7f6bbf-serving-cert\") pod \"service-ca-operator-7695fc6c8c-tcnzs\" (UID: \"22b39ae9-d2af-4874-9225-bd9a7a7f6bbf\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs"
Jul 16 04:54:18.448478 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.448459    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9582f0df-040b-4ff8-8eca-53207122af5b-serving-cert\") pod \"console-operator-7866497494-msc9n\" (UID: \"9582f0df-040b-4ff8-8eca-53207122af5b\") " pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:18.456428 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.456387    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwwvr\" (UniqueName: \"kubernetes.io/projected/22b39ae9-d2af-4874-9225-bd9a7a7f6bbf-kube-api-access-kwwvr\") pod \"service-ca-operator-7695fc6c8c-tcnzs\" (UID: \"22b39ae9-d2af-4874-9225-bd9a7a7f6bbf\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs"
Jul 16 04:54:18.456569 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.456509    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc7ps\" (UniqueName: \"kubernetes.io/projected/9582f0df-040b-4ff8-8eca-53207122af5b-kube-api-access-hc7ps\") pod \"console-operator-7866497494-msc9n\" (UID: \"9582f0df-040b-4ff8-8eca-53207122af5b\") " pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:18.456657 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.456609    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqnq6\" (UniqueName: \"kubernetes.io/projected/ba263a6a-62d7-412e-9459-04f53586cdbb-kube-api-access-vqnq6\") pod \"cluster-monitoring-operator-5f7f494dd4-k6lpx\" (UID: \"ba263a6a-62d7-412e-9459-04f53586cdbb\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"
Jul 16 04:54:18.456891 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.456873    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q56n\" (UniqueName: \"kubernetes.io/projected/22f69360-ae6d-446f-9267-daba8e58ed2b-kube-api-access-4q56n\") pod \"cluster-samples-operator-65db858894-cz8w7\" (UID: \"22f69360-ae6d-446f-9267-daba8e58ed2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7"
Jul 16 04:54:18.547270 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.547181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95bf5f5b-d8bb-4638-895a-015e35aea09a-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-2xgh8\" (UID: \"95bf5f5b-d8bb-4638-895a-015e35aea09a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8"
Jul 16 04:54:18.547270 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.547220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95bf5f5b-d8bb-4638-895a-015e35aea09a-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-2xgh8\" (UID: \"95bf5f5b-d8bb-4638-895a-015e35aea09a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8"
Jul 16 04:54:18.547496 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.547308    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggz77\" (UniqueName: \"kubernetes.io/projected/95bf5f5b-d8bb-4638-895a-015e35aea09a-kube-api-access-ggz77\") pod \"kube-storage-version-migrator-operator-799cb7f958-2xgh8\" (UID: \"95bf5f5b-d8bb-4638-895a-015e35aea09a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8"
Jul 16 04:54:18.547783 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.547761    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95bf5f5b-d8bb-4638-895a-015e35aea09a-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-2xgh8\" (UID: \"95bf5f5b-d8bb-4638-895a-015e35aea09a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8"
Jul 16 04:54:18.549768 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.549742    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95bf5f5b-d8bb-4638-895a-015e35aea09a-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-2xgh8\" (UID: \"95bf5f5b-d8bb-4638-895a-015e35aea09a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8"
Jul 16 04:54:18.554924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.554899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggz77\" (UniqueName: \"kubernetes.io/projected/95bf5f5b-d8bb-4638-895a-015e35aea09a-kube-api-access-ggz77\") pod \"kube-storage-version-migrator-operator-799cb7f958-2xgh8\" (UID: \"95bf5f5b-d8bb-4638-895a-015e35aea09a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8"
Jul 16 04:54:18.579866 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.579822    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs"
Jul 16 04:54:18.598508 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.598479    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:18.673198 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.672919    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8"
Jul 16 04:54:18.730544 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.730507    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs"]
Jul 16 04:54:18.735878 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:54:18.735844    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b39ae9_d2af_4874_9225_bd9a7a7f6bbf.slice/crio-072df9ffc49f2f1ec74240b9a8509dae5a110b11f8d7b6c68e39e42d8d865bde WatchSource:0}: Error finding container 072df9ffc49f2f1ec74240b9a8509dae5a110b11f8d7b6c68e39e42d8d865bde: Status 404 returned error can't find the container with id 072df9ffc49f2f1ec74240b9a8509dae5a110b11f8d7b6c68e39e42d8d865bde
Jul 16 04:54:18.744457 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.744388    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-msc9n"]
Jul 16 04:54:18.747751 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:54:18.747725    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9582f0df_040b_4ff8_8eca_53207122af5b.slice/crio-bc262852ab6d7f4d51a37e889c4e10c2c48d65a64a21ccf26145f64471c376ce WatchSource:0}: Error finding container bc262852ab6d7f4d51a37e889c4e10c2c48d65a64a21ccf26145f64471c376ce: Status 404 returned error can't find the container with id bc262852ab6d7f4d51a37e889c4e10c2c48d65a64a21ccf26145f64471c376ce
Jul 16 04:54:18.814813 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.814732    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8"]
Jul 16 04:54:18.818657 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:54:18.818630    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95bf5f5b_d8bb_4638_895a_015e35aea09a.slice/crio-47b203d704887b117bfb4907efc942c95cba64056654be67e1d3013991a75dd0 WatchSource:0}: Error finding container 47b203d704887b117bfb4907efc942c95cba64056654be67e1d3013991a75dd0: Status 404 returned error can't find the container with id 47b203d704887b117bfb4907efc942c95cba64056654be67e1d3013991a75dd0
Jul 16 04:54:18.926456 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.926420    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-msc9n" event={"ID":"9582f0df-040b-4ff8-8eca-53207122af5b","Type":"ContainerStarted","Data":"bc262852ab6d7f4d51a37e889c4e10c2c48d65a64a21ccf26145f64471c376ce"}
Jul 16 04:54:18.927423 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.927398    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs" event={"ID":"22b39ae9-d2af-4874-9225-bd9a7a7f6bbf","Type":"ContainerStarted","Data":"072df9ffc49f2f1ec74240b9a8509dae5a110b11f8d7b6c68e39e42d8d865bde"}
Jul 16 04:54:18.928432 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.928405    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8" event={"ID":"95bf5f5b-d8bb-4638-895a-015e35aea09a","Type":"ContainerStarted","Data":"47b203d704887b117bfb4907efc942c95cba64056654be67e1d3013991a75dd0"}
Jul 16 04:54:18.955333 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.955283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-k6lpx\" (UID: \"ba263a6a-62d7-412e-9459-04f53586cdbb\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"
Jul 16 04:54:18.955523 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:18.955377    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-cz8w7\" (UID: \"22f69360-ae6d-446f-9267-daba8e58ed2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7"
Jul 16 04:54:18.955523 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:18.955395    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 04:54:18.955523 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:18.955511    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls podName:ba263a6a-62d7-412e-9459-04f53586cdbb nodeName:}" failed. No retries permitted until 2026-07-16 04:54:19.955491793 +0000 UTC m=+263.371446843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-k6lpx" (UID: "ba263a6a-62d7-412e-9459-04f53586cdbb") : secret "cluster-monitoring-operator-tls" not found
Jul 16 04:54:18.955999 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:18.955982    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 04:54:18.956074 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:18.956057    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls podName:22f69360-ae6d-446f-9267-daba8e58ed2b nodeName:}" failed. No retries permitted until 2026-07-16 04:54:19.956033092 +0000 UTC m=+263.371988145 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls") pod "cluster-samples-operator-65db858894-cz8w7" (UID: "22f69360-ae6d-446f-9267-daba8e58ed2b") : secret "samples-operator-tls" not found
Jul 16 04:54:19.966128 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:19.966020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-k6lpx\" (UID: \"ba263a6a-62d7-412e-9459-04f53586cdbb\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"
Jul 16 04:54:19.966128 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:19.966083    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-cz8w7\" (UID: \"22f69360-ae6d-446f-9267-daba8e58ed2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7"
Jul 16 04:54:19.966750 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:19.966183    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 04:54:19.966750 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:19.966199    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 04:54:19.966750 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:19.966262    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls podName:ba263a6a-62d7-412e-9459-04f53586cdbb nodeName:}" failed. No retries permitted until 2026-07-16 04:54:21.966241283 +0000 UTC m=+265.382196335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-k6lpx" (UID: "ba263a6a-62d7-412e-9459-04f53586cdbb") : secret "cluster-monitoring-operator-tls" not found
Jul 16 04:54:19.966750 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:19.966281    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls podName:22f69360-ae6d-446f-9267-daba8e58ed2b nodeName:}" failed. No retries permitted until 2026-07-16 04:54:21.966272012 +0000 UTC m=+265.382227063 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls") pod "cluster-samples-operator-65db858894-cz8w7" (UID: "22f69360-ae6d-446f-9267-daba8e58ed2b") : secret "samples-operator-tls" not found
Jul 16 04:54:21.470693 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.470658    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8pp6l"]
Jul 16 04:54:21.475417 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.475389    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:21.478152 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.478124    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 04:54:21.478312 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.478126    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 04:54:21.478312 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.478128    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 04:54:21.479313 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.479291    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 04:54:21.479437 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.479291    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q2w4b\""
Jul 16 04:54:21.487283 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.487253    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8pp6l"]
Jul 16 04:54:21.580754 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.580721    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a68f257-3410-4b65-b926-58ff206520a5-data-volume\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:21.580966 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.580777    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbmwh\" (UniqueName: \"kubernetes.io/projected/7a68f257-3410-4b65-b926-58ff206520a5-kube-api-access-xbmwh\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:21.580966 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.580807    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:21.580966 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.580830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a68f257-3410-4b65-b926-58ff206520a5-crio-socket\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:21.580966 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.580901    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:21.681655 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.681611    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:21.681655 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.681660    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a68f257-3410-4b65-b926-58ff206520a5-crio-socket\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:21.681910 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.681701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:21.681910 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:21.681763    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 04:54:21.681910 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.681821    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a68f257-3410-4b65-b926-58ff206520a5-data-volume\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:21.681910 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.681845    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a68f257-3410-4b65-b926-58ff206520a5-crio-socket\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:21.681910 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:21.681860    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm podName:7a68f257-3410-4b65-b926-58ff206520a5 nodeName:}" failed. No retries permitted until 2026-07-16 04:54:22.181836837 +0000 UTC m=+265.597791890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8pp6l" (UID: "7a68f257-3410-4b65-b926-58ff206520a5") : configmap "kube-rbac-proxy" not found
Jul 16 04:54:21.681910 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.681900    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbmwh\" (UniqueName: \"kubernetes.io/projected/7a68f257-3410-4b65-b926-58ff206520a5-kube-api-access-xbmwh\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:21.682201 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:21.681929    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 04:54:21.682201 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:21.681971    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls podName:7a68f257-3410-4b65-b926-58ff206520a5 nodeName:}" failed. No retries permitted until 2026-07-16 04:54:22.181957497 +0000 UTC m=+265.597912550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8pp6l" (UID: "7a68f257-3410-4b65-b926-58ff206520a5") : secret "insights-runtime-extractor-tls" not found
Jul 16 04:54:21.682201 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.682164    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a68f257-3410-4b65-b926-58ff206520a5-data-volume\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:21.691268 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.691235    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbmwh\" (UniqueName: \"kubernetes.io/projected/7a68f257-3410-4b65-b926-58ff206520a5-kube-api-access-xbmwh\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:21.984447 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.984409    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-k6lpx\" (UID: \"ba263a6a-62d7-412e-9459-04f53586cdbb\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"
Jul 16 04:54:21.984602 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:21.984490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-cz8w7\" (UID: \"22f69360-ae6d-446f-9267-daba8e58ed2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7"
Jul 16 04:54:21.984602 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:21.984556    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 04:54:21.984717 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:21.984678    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 04:54:21.984717 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:21.984705    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls podName:ba263a6a-62d7-412e-9459-04f53586cdbb nodeName:}" failed. No retries permitted until 2026-07-16 04:54:25.984687447 +0000 UTC m=+269.400642498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-k6lpx" (UID: "ba263a6a-62d7-412e-9459-04f53586cdbb") : secret "cluster-monitoring-operator-tls" not found
Jul 16 04:54:21.984827 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:21.984731    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls podName:22f69360-ae6d-446f-9267-daba8e58ed2b nodeName:}" failed. No retries permitted until 2026-07-16 04:54:25.98471828 +0000 UTC m=+269.400673330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls") pod "cluster-samples-operator-65db858894-cz8w7" (UID: "22f69360-ae6d-446f-9267-daba8e58ed2b") : secret "samples-operator-tls" not found
Jul 16 04:54:22.186736 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:22.186689    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:22.186941 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:22.186826    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 04:54:22.186941 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:22.186858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:22.186941 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:22.186896    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls podName:7a68f257-3410-4b65-b926-58ff206520a5 nodeName:}" failed. No retries permitted until 2026-07-16 04:54:23.186874337 +0000 UTC m=+266.602829395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8pp6l" (UID: "7a68f257-3410-4b65-b926-58ff206520a5") : secret "insights-runtime-extractor-tls" not found
Jul 16 04:54:22.187108 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:22.186976    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 04:54:22.187108 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:22.187029    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm podName:7a68f257-3410-4b65-b926-58ff206520a5 nodeName:}" failed. No retries permitted until 2026-07-16 04:54:23.187016665 +0000 UTC m=+266.602971716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8pp6l" (UID: "7a68f257-3410-4b65-b926-58ff206520a5") : configmap "kube-rbac-proxy" not found
Jul 16 04:54:22.941269 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:22.941235    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/0.log"
Jul 16 04:54:22.941746 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:22.941274    2565 generic.go:356] "Generic (PLEG): container finished" podID="9582f0df-040b-4ff8-8eca-53207122af5b" containerID="83ce6155f5e974b2df3d9e211ca7932a4ff0aa3b150cff6768d87848379f2b00" exitCode=255
Jul 16 04:54:22.941746 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:22.941354    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-msc9n" event={"ID":"9582f0df-040b-4ff8-8eca-53207122af5b","Type":"ContainerDied","Data":"83ce6155f5e974b2df3d9e211ca7932a4ff0aa3b150cff6768d87848379f2b00"}
Jul 16 04:54:22.941746 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:22.941632    2565 scope.go:117] "RemoveContainer" containerID="83ce6155f5e974b2df3d9e211ca7932a4ff0aa3b150cff6768d87848379f2b00"
Jul 16 04:54:22.943073 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:22.943041    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs" event={"ID":"22b39ae9-d2af-4874-9225-bd9a7a7f6bbf","Type":"ContainerStarted","Data":"6ad908f8f4c262ed356b9c2b958a52d3d9a69f18a487b5f4b0a21637a85e9394"}
Jul 16 04:54:22.944639 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:22.944612    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8" event={"ID":"95bf5f5b-d8bb-4638-895a-015e35aea09a","Type":"ContainerStarted","Data":"ce189da53d6814803eedf53dc011d8df64fcd45ebbc697b2a9226671157fbbd1"}
Jul 16 04:54:22.972128 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:22.972074    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8" podStartSLOduration=1.829894804 podStartE2EDuration="4.972053974s" totalImagesPullingTime="3.14215917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:54:22.971052444 +0000 UTC m=+266.387007530" watchObservedRunningTime="2026-07-16 04:54:22.972053974 +0000 UTC m=+266.388009025"
Jul 16 04:54:22.986261 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:22.986062    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs" podStartSLOduration=1.7643533580000001 podStartE2EDuration="4.986038808s" totalImagesPullingTime="3.22168545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:54:22.985938782 +0000 UTC m=+266.401893855" watchObservedRunningTime="2026-07-16 04:54:22.986038808 +0000 UTC m=+266.401993881"
Jul 16 04:54:23.196581 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:23.196464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:23.196581 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:23.196537    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:23.196788 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:23.196619    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 04:54:23.196788 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:23.196692    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 04:54:23.196788 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:23.196698    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm podName:7a68f257-3410-4b65-b926-58ff206520a5 nodeName:}" failed. No retries permitted until 2026-07-16 04:54:25.196678698 +0000 UTC m=+268.612633754 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8pp6l" (UID: "7a68f257-3410-4b65-b926-58ff206520a5") : configmap "kube-rbac-proxy" not found
Jul 16 04:54:23.196788 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:23.196752    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls podName:7a68f257-3410-4b65-b926-58ff206520a5 nodeName:}" failed. No retries permitted until 2026-07-16 04:54:25.196738856 +0000 UTC m=+268.612693909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8pp6l" (UID: "7a68f257-3410-4b65-b926-58ff206520a5") : secret "insights-runtime-extractor-tls" not found
Jul 16 04:54:23.462116 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:23.462033    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cbrhk_4d3ed278-258b-4590-846b-c11283cbbba3/dns-node-resolver/0.log"
Jul 16 04:54:23.949275 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:23.949237    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 04:54:23.949874 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:23.949783    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/0.log"
Jul 16 04:54:23.949874 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:23.949828    2565 generic.go:356] "Generic (PLEG): container finished" podID="9582f0df-040b-4ff8-8eca-53207122af5b" containerID="881b66bad8f42fbe6e21b2cd1fdcaae362c9e915447a04de3e4e52b8654d9f7e" exitCode=255
Jul 16 04:54:23.949983 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:23.949916    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-msc9n" event={"ID":"9582f0df-040b-4ff8-8eca-53207122af5b","Type":"ContainerDied","Data":"881b66bad8f42fbe6e21b2cd1fdcaae362c9e915447a04de3e4e52b8654d9f7e"}
Jul 16 04:54:23.950029 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:23.949995    2565 scope.go:117] "RemoveContainer" containerID="83ce6155f5e974b2df3d9e211ca7932a4ff0aa3b150cff6768d87848379f2b00"
Jul 16 04:54:23.950278 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:23.950250    2565 scope.go:117] "RemoveContainer" containerID="881b66bad8f42fbe6e21b2cd1fdcaae362c9e915447a04de3e4e52b8654d9f7e"
Jul 16 04:54:23.950491 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:23.950460    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-msc9n_openshift-console-operator(9582f0df-040b-4ff8-8eca-53207122af5b)\"" pod="openshift-console-operator/console-operator-7866497494-msc9n" podUID="9582f0df-040b-4ff8-8eca-53207122af5b"
Jul 16 04:54:24.662919 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:24.662888    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sslfh_a2066081-de65-45f1-a46c-0f456249f49e/node-ca/0.log"
Jul 16 04:54:24.954957 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:24.954882    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 04:54:24.955309 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:24.955215    2565 scope.go:117] "RemoveContainer" containerID="881b66bad8f42fbe6e21b2cd1fdcaae362c9e915447a04de3e4e52b8654d9f7e"
Jul 16 04:54:24.955384 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:24.955363    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-msc9n_openshift-console-operator(9582f0df-040b-4ff8-8eca-53207122af5b)\"" pod="openshift-console-operator/console-operator-7866497494-msc9n" podUID="9582f0df-040b-4ff8-8eca-53207122af5b"
Jul 16 04:54:25.212369 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.212264    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:25.212369 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.212328    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:25.212630 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:25.212404    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 04:54:25.212630 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:25.212458    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 04:54:25.212630 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:25.212489    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm podName:7a68f257-3410-4b65-b926-58ff206520a5 nodeName:}" failed. No retries permitted until 2026-07-16 04:54:29.21247026 +0000 UTC m=+272.628425314 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8pp6l" (UID: "7a68f257-3410-4b65-b926-58ff206520a5") : configmap "kube-rbac-proxy" not found
Jul 16 04:54:25.212630 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:25.212512    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls podName:7a68f257-3410-4b65-b926-58ff206520a5 nodeName:}" failed. No retries permitted until 2026-07-16 04:54:29.212497342 +0000 UTC m=+272.628452400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8pp6l" (UID: "7a68f257-3410-4b65-b926-58ff206520a5") : secret "insights-runtime-extractor-tls" not found
Jul 16 04:54:25.262422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.262388    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dk6m5_31176ca2-45ab-4eb0-abc7-b4ceae534396/serve-healthcheck-canary/0.log"
Jul 16 04:54:25.454481 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.454446    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-szfxq"]
Jul 16 04:54:25.459141 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.459114    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-szfxq"
Jul 16 04:54:25.461542 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.461519    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 16 04:54:25.461720 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.461525    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 16 04:54:25.462538 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.462484    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 16 04:54:25.462538 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.462517    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 16 04:54:25.462671 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.462488    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-g5gkn\""
Jul 16 04:54:25.465051 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.465029    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-szfxq"]
Jul 16 04:54:25.514775 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.514736    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b8df92ce-3dbb-4091-ba92-9eff65b699bf-signing-cabundle\") pod \"service-ca-79f68b9f4f-szfxq\" (UID: \"b8df92ce-3dbb-4091-ba92-9eff65b699bf\") " pod="openshift-service-ca/service-ca-79f68b9f4f-szfxq"
Jul 16 04:54:25.514986 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.514894    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9kb8\" (UniqueName: \"kubernetes.io/projected/b8df92ce-3dbb-4091-ba92-9eff65b699bf-kube-api-access-x9kb8\") pod \"service-ca-79f68b9f4f-szfxq\" (UID: \"b8df92ce-3dbb-4091-ba92-9eff65b699bf\") " pod="openshift-service-ca/service-ca-79f68b9f4f-szfxq"
Jul 16 04:54:25.514986 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.514942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b8df92ce-3dbb-4091-ba92-9eff65b699bf-signing-key\") pod \"service-ca-79f68b9f4f-szfxq\" (UID: \"b8df92ce-3dbb-4091-ba92-9eff65b699bf\") " pod="openshift-service-ca/service-ca-79f68b9f4f-szfxq"
Jul 16 04:54:25.615409 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.615371    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b8df92ce-3dbb-4091-ba92-9eff65b699bf-signing-cabundle\") pod \"service-ca-79f68b9f4f-szfxq\" (UID: \"b8df92ce-3dbb-4091-ba92-9eff65b699bf\") " pod="openshift-service-ca/service-ca-79f68b9f4f-szfxq"
Jul 16 04:54:25.615572 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.615462    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9kb8\" (UniqueName: \"kubernetes.io/projected/b8df92ce-3dbb-4091-ba92-9eff65b699bf-kube-api-access-x9kb8\") pod \"service-ca-79f68b9f4f-szfxq\" (UID: \"b8df92ce-3dbb-4091-ba92-9eff65b699bf\") " pod="openshift-service-ca/service-ca-79f68b9f4f-szfxq"
Jul 16 04:54:25.615572 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.615489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b8df92ce-3dbb-4091-ba92-9eff65b699bf-signing-key\") pod \"service-ca-79f68b9f4f-szfxq\" (UID: \"b8df92ce-3dbb-4091-ba92-9eff65b699bf\") " pod="openshift-service-ca/service-ca-79f68b9f4f-szfxq"
Jul 16 04:54:25.616158 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.616137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b8df92ce-3dbb-4091-ba92-9eff65b699bf-signing-cabundle\") pod \"service-ca-79f68b9f4f-szfxq\" (UID: \"b8df92ce-3dbb-4091-ba92-9eff65b699bf\") " pod="openshift-service-ca/service-ca-79f68b9f4f-szfxq"
Jul 16 04:54:25.618109 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.618072    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b8df92ce-3dbb-4091-ba92-9eff65b699bf-signing-key\") pod \"service-ca-79f68b9f4f-szfxq\" (UID: \"b8df92ce-3dbb-4091-ba92-9eff65b699bf\") " pod="openshift-service-ca/service-ca-79f68b9f4f-szfxq"
Jul 16 04:54:25.623518 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.623493    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9kb8\" (UniqueName: \"kubernetes.io/projected/b8df92ce-3dbb-4091-ba92-9eff65b699bf-kube-api-access-x9kb8\") pod \"service-ca-79f68b9f4f-szfxq\" (UID: \"b8df92ce-3dbb-4091-ba92-9eff65b699bf\") " pod="openshift-service-ca/service-ca-79f68b9f4f-szfxq"
Jul 16 04:54:25.769379 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.769334    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-szfxq"
Jul 16 04:54:25.900053 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.900009    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-szfxq"]
Jul 16 04:54:25.903200 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:54:25.903168    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8df92ce_3dbb_4091_ba92_9eff65b699bf.slice/crio-9fc7e850bccc3a12d10e8c9b7f44f40dce013ae8ae7964cff9251c9125f66cd8 WatchSource:0}: Error finding container 9fc7e850bccc3a12d10e8c9b7f44f40dce013ae8ae7964cff9251c9125f66cd8: Status 404 returned error can't find the container with id 9fc7e850bccc3a12d10e8c9b7f44f40dce013ae8ae7964cff9251c9125f66cd8
Jul 16 04:54:25.958613 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:25.958567    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-szfxq" event={"ID":"b8df92ce-3dbb-4091-ba92-9eff65b699bf","Type":"ContainerStarted","Data":"9fc7e850bccc3a12d10e8c9b7f44f40dce013ae8ae7964cff9251c9125f66cd8"}
Jul 16 04:54:26.018894 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:26.018844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-cz8w7\" (UID: \"22f69360-ae6d-446f-9267-daba8e58ed2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7"
Jul 16 04:54:26.019074 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:26.018991    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-k6lpx\" (UID: \"ba263a6a-62d7-412e-9459-04f53586cdbb\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"
Jul 16 04:54:26.019074 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:26.019027    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 04:54:26.019157 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:26.019078    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 04:54:26.019157 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:26.019109    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls podName:22f69360-ae6d-446f-9267-daba8e58ed2b nodeName:}" failed. No retries permitted until 2026-07-16 04:54:34.019088449 +0000 UTC m=+277.435043508 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls") pod "cluster-samples-operator-65db858894-cz8w7" (UID: "22f69360-ae6d-446f-9267-daba8e58ed2b") : secret "samples-operator-tls" not found
Jul 16 04:54:26.019157 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:26.019127    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls podName:ba263a6a-62d7-412e-9459-04f53586cdbb nodeName:}" failed. No retries permitted until 2026-07-16 04:54:34.019115901 +0000 UTC m=+277.435070951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-k6lpx" (UID: "ba263a6a-62d7-412e-9459-04f53586cdbb") : secret "cluster-monitoring-operator-tls" not found
Jul 16 04:54:26.965003 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:26.964965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-szfxq" event={"ID":"b8df92ce-3dbb-4091-ba92-9eff65b699bf","Type":"ContainerStarted","Data":"de3d4a49580c3a23b5b39519eb2d9faad1c34665b2c1527b60f3dfb54fc4c981"}
Jul 16 04:54:26.981577 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:26.981513    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-szfxq" podStartSLOduration=1.9814946450000002 podStartE2EDuration="1.981494645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:54:26.98124978 +0000 UTC m=+270.397204854" watchObservedRunningTime="2026-07-16 04:54:26.981494645 +0000 UTC m=+270.397449718"
Jul 16 04:54:28.598961 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:28.598859    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:28.598961 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:28.598931    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:28.599362 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:28.599331    2565 scope.go:117] "RemoveContainer" containerID="881b66bad8f42fbe6e21b2cd1fdcaae362c9e915447a04de3e4e52b8654d9f7e"
Jul 16 04:54:28.599504 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:28.599484    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-msc9n_openshift-console-operator(9582f0df-040b-4ff8-8eca-53207122af5b)\"" pod="openshift-console-operator/console-operator-7866497494-msc9n" podUID="9582f0df-040b-4ff8-8eca-53207122af5b"
Jul 16 04:54:29.248919 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:29.248877    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:29.249111 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:29.249014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:29.249111 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:29.249045    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 04:54:29.249111 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:29.249088    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 04:54:29.249211 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:29.249118    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls podName:7a68f257-3410-4b65-b926-58ff206520a5 nodeName:}" failed. No retries permitted until 2026-07-16 04:54:37.249100752 +0000 UTC m=+280.665055803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8pp6l" (UID: "7a68f257-3410-4b65-b926-58ff206520a5") : secret "insights-runtime-extractor-tls" not found
Jul 16 04:54:29.249211 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:29.249136    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm podName:7a68f257-3410-4b65-b926-58ff206520a5 nodeName:}" failed. No retries permitted until 2026-07-16 04:54:37.249125136 +0000 UTC m=+280.665080187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8pp6l" (UID: "7a68f257-3410-4b65-b926-58ff206520a5") : configmap "kube-rbac-proxy" not found
Jul 16 04:54:34.094750 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:34.094702    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-k6lpx\" (UID: \"ba263a6a-62d7-412e-9459-04f53586cdbb\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"
Jul 16 04:54:34.094750 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:34.094758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-cz8w7\" (UID: \"22f69360-ae6d-446f-9267-daba8e58ed2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7"
Jul 16 04:54:34.097412 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:34.097380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba263a6a-62d7-412e-9459-04f53586cdbb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-k6lpx\" (UID: \"ba263a6a-62d7-412e-9459-04f53586cdbb\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"
Jul 16 04:54:34.097490 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:34.097428    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/22f69360-ae6d-446f-9267-daba8e58ed2b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-cz8w7\" (UID: \"22f69360-ae6d-446f-9267-daba8e58ed2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7"
Jul 16 04:54:34.185092 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:34.185052    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"
Jul 16 04:54:34.192314 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:34.192289    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7"
Jul 16 04:54:34.333170 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:34.333139    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx"]
Jul 16 04:54:34.336908 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:54:34.336876    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba263a6a_62d7_412e_9459_04f53586cdbb.slice/crio-31fb75c8e12fbf283ef682a43f04c46b4b574e95fec2bec022e40e9e8d66f5ba WatchSource:0}: Error finding container 31fb75c8e12fbf283ef682a43f04c46b4b574e95fec2bec022e40e9e8d66f5ba: Status 404 returned error can't find the container with id 31fb75c8e12fbf283ef682a43f04c46b4b574e95fec2bec022e40e9e8d66f5ba
Jul 16 04:54:34.351830 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:34.351748    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7"]
Jul 16 04:54:34.996572 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:34.996528    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7" event={"ID":"22f69360-ae6d-446f-9267-daba8e58ed2b","Type":"ContainerStarted","Data":"a3c0e2128ce688268b951d07235454199cf2a6b145c09effe00040b2bac0b186"}
Jul 16 04:54:34.997791 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:34.997757    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx" event={"ID":"ba263a6a-62d7-412e-9459-04f53586cdbb","Type":"ContainerStarted","Data":"31fb75c8e12fbf283ef682a43f04c46b4b574e95fec2bec022e40e9e8d66f5ba"}
Jul 16 04:54:37.007432 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:37.007396    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7" event={"ID":"22f69360-ae6d-446f-9267-daba8e58ed2b","Type":"ContainerStarted","Data":"5ce07605416cd0d17958cf29b751d33c8e07671ad63aa8ff2a0360d405dba5c3"}
Jul 16 04:54:37.007906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:37.007458    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7" event={"ID":"22f69360-ae6d-446f-9267-daba8e58ed2b","Type":"ContainerStarted","Data":"e51c20f83a10278053f000d83403728aa7709601895307ed266cb4a39a17082b"}
Jul 16 04:54:37.008984 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:37.008957    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx" event={"ID":"ba263a6a-62d7-412e-9459-04f53586cdbb","Type":"ContainerStarted","Data":"14d3a567dc9b27d09b38e08241b9981a46749d79af20aac85417efad0dc933bb"}
Jul 16 04:54:37.022107 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:37.022054    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cz8w7" podStartSLOduration=16.634991651 podStartE2EDuration="19.022030677s" totalImagesPullingTime="2.387039026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:54:37.021250813 +0000 UTC m=+280.437205886" watchObservedRunningTime="2026-07-16 04:54:37.022030677 +0000 UTC m=+280.437985751"
Jul 16 04:54:37.037198 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:37.037100    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-k6lpx" podStartSLOduration=16.596665533 podStartE2EDuration="19.037083961s" totalImagesPullingTime="2.440418428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:54:37.036270001 +0000 UTC m=+280.452225075" watchObservedRunningTime="2026-07-16 04:54:37.037083961 +0000 UTC m=+280.453039033"
Jul 16 04:54:37.325130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:37.325027    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:37.325130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:37.325091    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:37.325650 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:37.325629    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a68f257-3410-4b65-b926-58ff206520a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:37.327835 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:37.327813    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a68f257-3410-4b65-b926-58ff206520a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8pp6l\" (UID: \"7a68f257-3410-4b65-b926-58ff206520a5\") " pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:37.386732 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:37.386696    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8pp6l"
Jul 16 04:54:37.531845 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:37.531814    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8pp6l"]
Jul 16 04:54:37.534655 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:54:37.534630    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a68f257_3410_4b65_b926_58ff206520a5.slice/crio-83784e3d6cb3892d922502773de40df02408d1381bc413600743fe1c728673b7 WatchSource:0}: Error finding container 83784e3d6cb3892d922502773de40df02408d1381bc413600743fe1c728673b7: Status 404 returned error can't find the container with id 83784e3d6cb3892d922502773de40df02408d1381bc413600743fe1c728673b7
Jul 16 04:54:38.013219 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:38.013184    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8pp6l" event={"ID":"7a68f257-3410-4b65-b926-58ff206520a5","Type":"ContainerStarted","Data":"83784e3d6cb3892d922502773de40df02408d1381bc413600743fe1c728673b7"}
Jul 16 04:54:39.680574 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:39.680529    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd" podUID="40f2c277-95a8-4dc0-acda-1fd724cd9c0f"
Jul 16 04:54:39.680574 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:39.680529    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw" podUID="5dedf531-2032-458b-8003-a88d1379a808"
Jul 16 04:54:39.681106 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:39.680529    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tfrst" podUID="d2e8927f-33f5-4cce-a369-b7736e189b57"
Jul 16 04:54:40.020506 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:40.020419    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:54:40.020506 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:40.020419    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tfrst"
Jul 16 04:54:40.020808 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:40.020438    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:54:42.873479 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:42.873386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:54:42.873479 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:42.873432    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:54:42.873479 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:42.873459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ldqtw\" (UID: \"5dedf531-2032-458b-8003-a88d1379a808\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:54:42.876134 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:42.876102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2e8927f-33f5-4cce-a369-b7736e189b57-metrics-tls\") pod \"dns-default-tfrst\" (UID: \"d2e8927f-33f5-4cce-a369-b7736e189b57\") " pod="openshift-dns/dns-default-tfrst"
Jul 16 04:54:42.876265 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:42.876109    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls\") pod \"image-registry-776bb7cd46-j6xbd\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") " pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:54:42.876265 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:42.876117    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5dedf531-2032-458b-8003-a88d1379a808-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ldqtw\" (UID: \"5dedf531-2032-458b-8003-a88d1379a808\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:54:43.025426 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:43.024746    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-j9k26\""
Jul 16 04:54:43.025426 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:43.025068    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8whht\""
Jul 16 04:54:43.025426 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:43.025272    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dcm9d\""
Jul 16 04:54:43.031545 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:43.031284    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"
Jul 16 04:54:43.031978 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:43.031819    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tfrst"
Jul 16 04:54:43.031978 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:43.031901    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:54:43.036794 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:43.036477    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8pp6l" event={"ID":"7a68f257-3410-4b65-b926-58ff206520a5","Type":"ContainerStarted","Data":"7d62dafe6f7948a54ba97e22f374a17818b254692768458ce6c990a961a08255"}
Jul 16 04:54:43.136798 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:43.136380    2565 scope.go:117] "RemoveContainer" containerID="881b66bad8f42fbe6e21b2cd1fdcaae362c9e915447a04de3e4e52b8654d9f7e"
Jul 16 04:54:43.241514 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:43.241397    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tfrst"]
Jul 16 04:54:43.247942 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:54:43.247903    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e8927f_33f5_4cce_a369_b7736e189b57.slice/crio-5c1de746e626c8d180c95d537e3775ff7f966b212e90b5f64425a88aec63191f WatchSource:0}: Error finding container 5c1de746e626c8d180c95d537e3775ff7f966b212e90b5f64425a88aec63191f: Status 404 returned error can't find the container with id 5c1de746e626c8d180c95d537e3775ff7f966b212e90b5f64425a88aec63191f
Jul 16 04:54:43.474321 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:43.473857    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-ldqtw"]
Jul 16 04:54:43.475678 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:43.475652    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-776bb7cd46-j6xbd"]
Jul 16 04:54:43.477869 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:54:43.477835    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dedf531_2032_458b_8003_a88d1379a808.slice/crio-0f0f93074d5e83dbcde0d18c8ce84e1333e9ab734d9ea97623d9144bc0b573a3 WatchSource:0}: Error finding container 0f0f93074d5e83dbcde0d18c8ce84e1333e9ab734d9ea97623d9144bc0b573a3: Status 404 returned error can't find the container with id 0f0f93074d5e83dbcde0d18c8ce84e1333e9ab734d9ea97623d9144bc0b573a3
Jul 16 04:54:43.479096 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:54:43.479068    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f2c277_95a8_4dc0_acda_1fd724cd9c0f.slice/crio-a5a85e8276dc1d75cf510842d309a11f1c3b8efb63991bfcb620ab7d2691d50e WatchSource:0}: Error finding container a5a85e8276dc1d75cf510842d309a11f1c3b8efb63991bfcb620ab7d2691d50e: Status 404 returned error can't find the container with id a5a85e8276dc1d75cf510842d309a11f1c3b8efb63991bfcb620ab7d2691d50e
Jul 16 04:54:44.043512 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.043363    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd" event={"ID":"40f2c277-95a8-4dc0-acda-1fd724cd9c0f","Type":"ContainerStarted","Data":"7b9aa874980aa2f8d6af448d0705c1de3e95c78a649c6ac0a723f92a168fc10a"}
Jul 16 04:54:44.043512 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.043469    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd" event={"ID":"40f2c277-95a8-4dc0-acda-1fd724cd9c0f","Type":"ContainerStarted","Data":"a5a85e8276dc1d75cf510842d309a11f1c3b8efb63991bfcb620ab7d2691d50e"}
Jul 16 04:54:44.044064 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.043743    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:54:44.046140 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.046098    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw" event={"ID":"5dedf531-2032-458b-8003-a88d1379a808","Type":"ContainerStarted","Data":"0f0f93074d5e83dbcde0d18c8ce84e1333e9ab734d9ea97623d9144bc0b573a3"}
Jul 16 04:54:44.048880 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.048830    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 04:54:44.049023 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.048968    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-msc9n" event={"ID":"9582f0df-040b-4ff8-8eca-53207122af5b","Type":"ContainerStarted","Data":"ed7b8018d6e336d0e9b16a05b5b922ae19c47db60194632ac02cf90c2b2384c6"}
Jul 16 04:54:44.049648 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.049502    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:44.050997 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.050964    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tfrst" event={"ID":"d2e8927f-33f5-4cce-a369-b7736e189b57","Type":"ContainerStarted","Data":"5c1de746e626c8d180c95d537e3775ff7f966b212e90b5f64425a88aec63191f"}
Jul 16 04:54:44.060013 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.057658    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-msc9n"
Jul 16 04:54:44.065863 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.065784    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd" podStartSLOduration=286.065761723 podStartE2EDuration="4m46.065761723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:54:44.064404833 +0000 UTC m=+287.480359907" watchObservedRunningTime="2026-07-16 04:54:44.065761723 +0000 UTC m=+287.481716800"
Jul 16 04:54:44.080518 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.080055    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-msc9n" podStartSLOduration=22.863796065 podStartE2EDuration="26.080024674s" totalImagesPullingTime="3.216228609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:54:44.079705829 +0000 UTC m=+287.495660904" watchObservedRunningTime="2026-07-16 04:54:44.080024674 +0000 UTC m=+287.495979748"
Jul 16 04:54:44.484641 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.484580    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-w6c87"]
Jul 16 04:54:44.494794 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.494760    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-w6c87"
Jul 16 04:54:44.498049 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.497793    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 16 04:54:44.498049 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.497792    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rcrnx\""
Jul 16 04:54:44.499265 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.498945    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-wbzhk"]
Jul 16 04:54:44.503317 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.502851    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-w6c87"]
Jul 16 04:54:44.503317 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.502967    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-wbzhk"
Jul 16 04:54:44.506730 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.506701    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 16 04:54:44.507027 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.507010    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 16 04:54:44.507526 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.507449    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-nwdw6\""
Jul 16 04:54:44.514981 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.514481    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-wbzhk"]
Jul 16 04:54:44.592044 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.591854    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zk5v\" (UniqueName: \"kubernetes.io/projected/00dab318-0248-4bf4-b18b-d439c871101d-kube-api-access-7zk5v\") pod \"downloads-68c9cbb497-wbzhk\" (UID: \"00dab318-0248-4bf4-b18b-d439c871101d\") " pod="openshift-console/downloads-68c9cbb497-wbzhk"
Jul 16 04:54:44.592044 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.591903    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/84c30e52-23b6-4763-811a-1c81d8712e01-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-w6c87\" (UID: \"84c30e52-23b6-4763-811a-1c81d8712e01\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-w6c87"
Jul 16 04:54:44.693337 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.693301    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zk5v\" (UniqueName: \"kubernetes.io/projected/00dab318-0248-4bf4-b18b-d439c871101d-kube-api-access-7zk5v\") pod \"downloads-68c9cbb497-wbzhk\" (UID: \"00dab318-0248-4bf4-b18b-d439c871101d\") " pod="openshift-console/downloads-68c9cbb497-wbzhk"
Jul 16 04:54:44.693337 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.693340    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/84c30e52-23b6-4763-811a-1c81d8712e01-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-w6c87\" (UID: \"84c30e52-23b6-4763-811a-1c81d8712e01\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-w6c87"
Jul 16 04:54:44.696372 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.696343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/84c30e52-23b6-4763-811a-1c81d8712e01-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-w6c87\" (UID: \"84c30e52-23b6-4763-811a-1c81d8712e01\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-w6c87"
Jul 16 04:54:44.701767 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.701740    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zk5v\" (UniqueName: \"kubernetes.io/projected/00dab318-0248-4bf4-b18b-d439c871101d-kube-api-access-7zk5v\") pod \"downloads-68c9cbb497-wbzhk\" (UID: \"00dab318-0248-4bf4-b18b-d439c871101d\") " pod="openshift-console/downloads-68c9cbb497-wbzhk"
Jul 16 04:54:44.811758 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.811666    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-w6c87"
Jul 16 04:54:44.821030 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:44.820995    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-wbzhk"
Jul 16 04:54:47.282659 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:47.282345    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-wbzhk"]
Jul 16 04:54:47.286341 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:54:47.286296    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00dab318_0248_4bf4_b18b_d439c871101d.slice/crio-3ac78d841e433c15d5f6fa3f29b01c7987d4fb117c64caddcb9c2e2e2d3652c2 WatchSource:0}: Error finding container 3ac78d841e433c15d5f6fa3f29b01c7987d4fb117c64caddcb9c2e2e2d3652c2: Status 404 returned error can't find the container with id 3ac78d841e433c15d5f6fa3f29b01c7987d4fb117c64caddcb9c2e2e2d3652c2
Jul 16 04:54:47.303990 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:47.303961    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-w6c87"]
Jul 16 04:54:47.307051 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:54:47.307016    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c30e52_23b6_4763_811a_1c81d8712e01.slice/crio-a10bf4fdc1ee18c9386322554c5adb6c563b6fe4905e8ba7995b220bb6213461 WatchSource:0}: Error finding container a10bf4fdc1ee18c9386322554c5adb6c563b6fe4905e8ba7995b220bb6213461: Status 404 returned error can't find the container with id a10bf4fdc1ee18c9386322554c5adb6c563b6fe4905e8ba7995b220bb6213461
Jul 16 04:54:48.071062 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:48.070951    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw" event={"ID":"5dedf531-2032-458b-8003-a88d1379a808","Type":"ContainerStarted","Data":"2cb4f4418baa3b4cde15a63c6dc2b41a90db91e343b8a876e1dfe6821a0239a8"}
Jul 16 04:54:48.074371 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:48.074221    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8pp6l" event={"ID":"7a68f257-3410-4b65-b926-58ff206520a5","Type":"ContainerStarted","Data":"ae47089ccf1451ef759cd5f1e5ac4489848cb2a44adfefcc2b6afd08e2fccd2c"}
Jul 16 04:54:48.076663 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:48.076580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-w6c87" event={"ID":"84c30e52-23b6-4763-811a-1c81d8712e01","Type":"ContainerStarted","Data":"a10bf4fdc1ee18c9386322554c5adb6c563b6fe4905e8ba7995b220bb6213461"}
Jul 16 04:54:48.079522 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:48.079492    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tfrst" event={"ID":"d2e8927f-33f5-4cce-a369-b7736e189b57","Type":"ContainerStarted","Data":"2c11fc9050fc2baa21012b13ea8c9a86fd216e199bfb5e0823f9f3fbc1bf59fc"}
Jul 16 04:54:48.079678 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:48.079552    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tfrst" event={"ID":"d2e8927f-33f5-4cce-a369-b7736e189b57","Type":"ContainerStarted","Data":"de5d752842d1174847d553e5d77a002e1fffe46bd0793d533ff6c9e60528a6aa"}
Jul 16 04:54:48.080306 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:48.080284    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tfrst"
Jul 16 04:54:48.081992 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:48.081959    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-wbzhk" event={"ID":"00dab318-0248-4bf4-b18b-d439c871101d","Type":"ContainerStarted","Data":"3ac78d841e433c15d5f6fa3f29b01c7987d4fb117c64caddcb9c2e2e2d3652c2"}
Jul 16 04:54:48.086646 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:48.086577    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-ldqtw" podStartSLOduration=256.45409698 podStartE2EDuration="4m20.086538728s" totalImagesPullingTime="3.632441748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:54:48.08577273 +0000 UTC m=+291.501727804" watchObservedRunningTime="2026-07-16 04:54:48.086538728 +0000 UTC m=+291.502493804"
Jul 16 04:54:49.089135 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:49.089096    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-w6c87" event={"ID":"84c30e52-23b6-4763-811a-1c81d8712e01","Type":"ContainerStarted","Data":"f23511490856a4ee73078f971a95557a20d58181a54c464ed7b388fc26c1c815"}
Jul 16 04:54:49.089830 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:49.089775    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-w6c87"
Jul 16 04:54:49.096063 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:49.096014    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-w6c87"
Jul 16 04:54:49.104406 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:49.104350    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-w6c87" podStartSLOduration=3.850646254 podStartE2EDuration="5.104331526s" totalImagesPullingTime="1.253685272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:54:49.10320866 +0000 UTC m=+292.519163738" watchObservedRunningTime="2026-07-16 04:54:49.104331526 +0000 UTC m=+292.520286600"
Jul 16 04:54:49.104630 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:49.104555    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tfrst" podStartSLOduration=253.243496805 podStartE2EDuration="4m17.104543802s" totalImagesPullingTime="3.861046997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:54:48.103029578 +0000 UTC m=+291.518984650" watchObservedRunningTime="2026-07-16 04:54:49.104543802 +0000 UTC m=+292.520498879"
Jul 16 04:54:50.141442 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:50.141390    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ldxlj" podUID="c028e133-a290-4758-998c-24e158389825"
Jul 16 04:54:52.105403 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:52.105363    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8pp6l" event={"ID":"7a68f257-3410-4b65-b926-58ff206520a5","Type":"ContainerStarted","Data":"7e787f88038e54dc73e9402885803d626a1efa5159eb20235ebf437883471f34"}
Jul 16 04:54:52.123015 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:52.122959    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8pp6l" podStartSLOduration=17.113737069 podStartE2EDuration="31.122938761s" totalImagesPullingTime="14.009201692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:54:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:54:52.120829029 +0000 UTC m=+295.536784104" watchObservedRunningTime="2026-07-16 04:54:52.122938761 +0000 UTC m=+295.538893833"
Jul 16 04:54:52.961073 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:52.961031    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b8d7cf447-fxmqb"]
Jul 16 04:54:52.964646 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:52.964616    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:52.967524 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:52.967383    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 16 04:54:52.967524 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:52.967416    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 16 04:54:52.967524 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:52.967472    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4bwwt\""
Jul 16 04:54:52.968538 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:52.968516    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 16 04:54:52.968538 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:52.968534    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 16 04:54:52.968727 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:52.968534    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 16 04:54:52.972691 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:52.972662    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b8d7cf447-fxmqb"]
Jul 16 04:54:53.068577 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.068536    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-oauth-serving-cert\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.068786 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.068617    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-serving-cert\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.068786 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.068651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-oauth-config\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.068786 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.068679    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-config\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.068786 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.068709    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-service-ca\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.068786 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.068744    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w26s4\" (UniqueName: \"kubernetes.io/projected/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-kube-api-access-w26s4\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.169798 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.169738    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-oauth-serving-cert\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.169798 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.169801    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-serving-cert\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.170326 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.169876    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-oauth-config\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.170326 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.169903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-config\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.170326 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.169931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-service-ca\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.170326 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.169967    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w26s4\" (UniqueName: \"kubernetes.io/projected/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-kube-api-access-w26s4\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.171061 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.170697    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-oauth-serving-cert\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.171061 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.170900    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-service-ca\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.171289 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.171271    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-config\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.173011 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.172980    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-oauth-config\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.173154 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.173110    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-serving-cert\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.181542 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.181505    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w26s4\" (UniqueName: \"kubernetes.io/projected/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-kube-api-access-w26s4\") pod \"console-6b8d7cf447-fxmqb\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") " pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.277912 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.277873    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:54:53.437099 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:53.437069    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b8d7cf447-fxmqb"]
Jul 16 04:54:53.439740 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:54:53.439707    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afa39ff_fec3_4bcc_b1a3_c2cd92df7e43.slice/crio-3d63e720c038cdec9d0123067ccfaa1c8cccd7da6368d1a1539aa9efd10548f6 WatchSource:0}: Error finding container 3d63e720c038cdec9d0123067ccfaa1c8cccd7da6368d1a1539aa9efd10548f6: Status 404 returned error can't find the container with id 3d63e720c038cdec9d0123067ccfaa1c8cccd7da6368d1a1539aa9efd10548f6
Jul 16 04:54:54.115217 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:54.115177    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8d7cf447-fxmqb" event={"ID":"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43","Type":"ContainerStarted","Data":"3d63e720c038cdec9d0123067ccfaa1c8cccd7da6368d1a1539aa9efd10548f6"}
Jul 16 04:54:55.152158 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.152118    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5679894dd6-4gm5p"]
Jul 16 04:54:55.171971 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.171929    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5679894dd6-4gm5p"]
Jul 16 04:54:55.172142 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.172090    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.186226 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.186190    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 16 04:54:55.187194 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.187165    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-oauth-config\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.187387 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.187370    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rllz9\" (UniqueName: \"kubernetes.io/projected/1901fb00-4f59-47e2-ae36-d51a76acb59c-kube-api-access-rllz9\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.187501 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.187489    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-trusted-ca-bundle\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.187647 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.187624    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-config\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.187766 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.187660    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-service-ca\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.187766 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.187726    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-oauth-serving-cert\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.187866 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.187777    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-serving-cert\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.288611 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.288550    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-oauth-config\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.288611 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.288614    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rllz9\" (UniqueName: \"kubernetes.io/projected/1901fb00-4f59-47e2-ae36-d51a76acb59c-kube-api-access-rllz9\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.288871 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.288807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-trusted-ca-bundle\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.288928 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.288873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-config\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.288928 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.288898    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-service-ca\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.289031 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.288975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-oauth-serving-cert\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.289031 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.289022    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-serving-cert\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.289648 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.289622    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-config\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.290381 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.290353    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-service-ca\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.290754 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.290727    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-trusted-ca-bundle\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.290837 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.290790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-oauth-serving-cert\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.292705 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.292685    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-serving-cert\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.292819 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.292685    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-oauth-config\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.296450 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.296402    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rllz9\" (UniqueName: \"kubernetes.io/projected/1901fb00-4f59-47e2-ae36-d51a76acb59c-kube-api-access-rllz9\") pod \"console-5679894dd6-4gm5p\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") " pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.489402 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.489362    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:54:55.697220 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.697106    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5679894dd6-4gm5p"]
Jul 16 04:54:55.701839 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.701808    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"]
Jul 16 04:54:55.723363 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.723325    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"]
Jul 16 04:54:55.723881 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.723706    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.728847 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.728743    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 04:54:55.730921 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.729011    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 16 04:54:55.730921 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.729205    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 16 04:54:55.730921 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.729354    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6hgsv\""
Jul 16 04:54:55.730921 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.729390    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 16 04:54:55.789579 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.787907    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7rtjs"]
Jul 16 04:54:55.795257 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.794800    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea86cd3f-b598-443a-8d70-cb2fc310aecd-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.795257 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.794849    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.795257 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.794881    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.795257 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.794935    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x65f9\" (UniqueName: \"kubernetes.io/projected/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-api-access-x65f9\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.795257 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.794998    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.795257 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.795025    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea86cd3f-b598-443a-8d70-cb2fc310aecd-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.803045 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.803014    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.806606 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.806557    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m8fw4\""
Jul 16 04:54:55.806925 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.806903    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 04:54:55.807135 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.807115    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 04:54:55.808292 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.808275    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 04:54:55.896407 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.895673    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x65f9\" (UniqueName: \"kubernetes.io/projected/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-api-access-x65f9\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.896407 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.895729    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-wtmp\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.896407 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.895765    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.896407 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.895807    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-sys\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.896407 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.895842    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.896407 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.895868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea86cd3f-b598-443a-8d70-cb2fc310aecd-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.896407 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.895894    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx7s6\" (UniqueName: \"kubernetes.io/projected/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-kube-api-access-dx7s6\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.896407 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.895919    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-tls\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.896407 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.895946    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-textfile\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.896407 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.895994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea86cd3f-b598-443a-8d70-cb2fc310aecd-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.896407 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.896025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.896407 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.896050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.896407 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.896092    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.896407 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.896129    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-metrics-client-ca\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.896407 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.896154    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-root\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.897328 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.897118    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea86cd3f-b598-443a-8d70-cb2fc310aecd-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.897328 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.897144    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea86cd3f-b598-443a-8d70-cb2fc310aecd-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.897328 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:55.897316    2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 16 04:54:55.897473 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:55.897375    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-state-metrics-tls podName:ea86cd3f-b598-443a-8d70-cb2fc310aecd nodeName:}" failed. No retries permitted until 2026-07-16 04:54:56.397354675 +0000 UTC m=+299.813309739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-9zsxd" (UID: "ea86cd3f-b598-443a-8d70-cb2fc310aecd") : secret "kube-state-metrics-tls" not found
Jul 16 04:54:55.897756 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.897731    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.907529 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.907494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x65f9\" (UniqueName: \"kubernetes.io/projected/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-api-access-x65f9\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.909885 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.909831    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:55.998626 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.997441    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-root\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.998626 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.997547    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-wtmp\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.998626 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.997602    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.998626 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.997649    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-sys\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.998626 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.997701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx7s6\" (UniqueName: \"kubernetes.io/projected/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-kube-api-access-dx7s6\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.998626 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.997730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-tls\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.998626 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.997756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-textfile\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.998626 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.997833    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.998626 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.997874    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-metrics-client-ca\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.998626 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.998487    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-metrics-client-ca\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:55.998626 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:55.998556    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-root\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:56.001952 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:56.000652    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-textfile\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:56.001952 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:56.000759    2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 16 04:54:56.001952 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:56.000821    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-tls podName:1ce6ade3-c5ef-48dc-aec9-85277e8472fd nodeName:}" failed. No retries permitted until 2026-07-16 04:54:56.500799028 +0000 UTC m=+299.916754106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-tls") pod "node-exporter-7rtjs" (UID: "1ce6ade3-c5ef-48dc-aec9-85277e8472fd") : secret "node-exporter-tls" not found
Jul 16 04:54:56.001952 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:56.001144    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:56.001952 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:56.001039    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-sys\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:56.002823 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:56.002777    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-wtmp\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:56.003259 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:56.003217    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:56.014190 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:56.014120    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx7s6\" (UniqueName: \"kubernetes.io/projected/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-kube-api-access-dx7s6\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:56.402139 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:56.402092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:56.402685 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:56.402251    2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 16 04:54:56.402685 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:54:56.402324    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-state-metrics-tls podName:ea86cd3f-b598-443a-8d70-cb2fc310aecd nodeName:}" failed. No retries permitted until 2026-07-16 04:54:57.402308415 +0000 UTC m=+300.818263465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-9zsxd" (UID: "ea86cd3f-b598-443a-8d70-cb2fc310aecd") : secret "kube-state-metrics-tls" not found
Jul 16 04:54:56.502974 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:56.502934    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-tls\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:56.506648 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:56.506234    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1ce6ade3-c5ef-48dc-aec9-85277e8472fd-node-exporter-tls\") pod \"node-exporter-7rtjs\" (UID: \"1ce6ade3-c5ef-48dc-aec9-85277e8472fd\") " pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:56.719466 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:56.719300    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7rtjs"
Jul 16 04:54:57.262407 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:54:57.262368    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1901fb00_4f59_47e2_ae36_d51a76acb59c.slice/crio-5694c7385e5a401c5436ce97dd472075fcc6f7461375be006edbdbbc8b42ecfa WatchSource:0}: Error finding container 5694c7385e5a401c5436ce97dd472075fcc6f7461375be006edbdbbc8b42ecfa: Status 404 returned error can't find the container with id 5694c7385e5a401c5436ce97dd472075fcc6f7461375be006edbdbbc8b42ecfa
Jul 16 04:54:57.270513 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:54:57.270482    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce6ade3_c5ef_48dc_aec9_85277e8472fd.slice/crio-98d7e8cc7879511d41aad97b4f947fd0f49addcb53efd71f2384ae051ac69a61 WatchSource:0}: Error finding container 98d7e8cc7879511d41aad97b4f947fd0f49addcb53efd71f2384ae051ac69a61: Status 404 returned error can't find the container with id 98d7e8cc7879511d41aad97b4f947fd0f49addcb53efd71f2384ae051ac69a61
Jul 16 04:54:57.295861 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:57.295832    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 04:54:57.301332 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:57.301305    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 04:54:57.322029 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:57.322005    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 04:54:57.328611 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:57.328565    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 04:54:57.343660 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:57.343628    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 04:54:57.413171 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:57.413060    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:57.417552 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:57.417405    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea86cd3f-b598-443a-8d70-cb2fc310aecd-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-9zsxd\" (UID: \"ea86cd3f-b598-443a-8d70-cb2fc310aecd\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:57.542626 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:57.542511    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6hgsv\""
Jul 16 04:54:57.550420 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:57.550387    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"
Jul 16 04:54:57.715367 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:57.715213    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-9zsxd"]
Jul 16 04:54:57.718482 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:54:57.718452    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea86cd3f_b598_443a_8d70_cb2fc310aecd.slice/crio-b9b248518316ca924cd6409a3552c89b4f84f58822ced5b1b04be3b89e956309 WatchSource:0}: Error finding container b9b248518316ca924cd6409a3552c89b4f84f58822ced5b1b04be3b89e956309: Status 404 returned error can't find the container with id b9b248518316ca924cd6409a3552c89b4f84f58822ced5b1b04be3b89e956309
Jul 16 04:54:58.133496 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.133455    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd" event={"ID":"ea86cd3f-b598-443a-8d70-cb2fc310aecd","Type":"ContainerStarted","Data":"b9b248518316ca924cd6409a3552c89b4f84f58822ced5b1b04be3b89e956309"}
Jul 16 04:54:58.134860 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.134824    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7rtjs" event={"ID":"1ce6ade3-c5ef-48dc-aec9-85277e8472fd","Type":"ContainerStarted","Data":"98d7e8cc7879511d41aad97b4f947fd0f49addcb53efd71f2384ae051ac69a61"}
Jul 16 04:54:58.136792 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.136756    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8d7cf447-fxmqb" event={"ID":"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43","Type":"ContainerStarted","Data":"8cfe76e172bb39b05faefb48914cd0e6eeffa9ca334ded6fc30783a42bdc3a62"}
Jul 16 04:54:58.138720 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.138680    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5679894dd6-4gm5p" event={"ID":"1901fb00-4f59-47e2-ae36-d51a76acb59c","Type":"ContainerStarted","Data":"5d078b011d316a061e31e1f93d5477c6778faa55b325050d950fe607e2607a4e"}
Jul 16 04:54:58.138905 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.138747    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5679894dd6-4gm5p" event={"ID":"1901fb00-4f59-47e2-ae36-d51a76acb59c","Type":"ContainerStarted","Data":"5694c7385e5a401c5436ce97dd472075fcc6f7461375be006edbdbbc8b42ecfa"}
Jul 16 04:54:58.153638 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.153549    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b8d7cf447-fxmqb" podStartSLOduration=2.271272817 podStartE2EDuration="6.153527214s" totalImagesPullingTime="3.882254397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:54:58.153117491 +0000 UTC m=+301.569072564" watchObservedRunningTime="2026-07-16 04:54:58.153527214 +0000 UTC m=+301.569482287"
Jul 16 04:54:58.168643 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.168560    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5679894dd6-4gm5p" podStartSLOduration=2.807590195 podStartE2EDuration="3.168538466s" totalImagesPullingTime="360.948271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:54:58.168471065 +0000 UTC m=+301.584426138" watchObservedRunningTime="2026-07-16 04:54:58.168538466 +0000 UTC m=+301.584493596"
Jul 16 04:54:58.687949 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.687910    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7c68d77877-fkzdg"]
Jul 16 04:54:58.693211 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.693180    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.695934 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.695906    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bv5alvmcak3l3\""
Jul 16 04:54:58.696108 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.695962    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-t2jk6\""
Jul 16 04:54:58.696108 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.695909    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 16 04:54:58.696108 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.695906    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 16 04:54:58.696281 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.696036    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 16 04:54:58.696281 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.696203    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 16 04:54:58.696450 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.696427    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 16 04:54:58.703470 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.703421    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c68d77877-fkzdg"]
Jul 16 04:54:58.724647 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.724214    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-thanos-querier-tls\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.724647 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.724268    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.724647 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.724306    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-grpc-tls\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.724647 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.724393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.724647 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.724417    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcz5s\" (UniqueName: \"kubernetes.io/projected/4a6fa390-9d6a-4590-97a0-baa59f13b954-kube-api-access-xcz5s\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.724647 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.724446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.724647 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.724478    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.724647 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.724514    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a6fa390-9d6a-4590-97a0-baa59f13b954-metrics-client-ca\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.825393 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.825352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-thanos-querier-tls\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.825623 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.825409    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.825623 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.825444    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-grpc-tls\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.825623 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.825524    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.825623 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.825547    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcz5s\" (UniqueName: \"kubernetes.io/projected/4a6fa390-9d6a-4590-97a0-baa59f13b954-kube-api-access-xcz5s\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.825623 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.825577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.825623 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.825623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.825936 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.825656    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a6fa390-9d6a-4590-97a0-baa59f13b954-metrics-client-ca\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.826541 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.826445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a6fa390-9d6a-4590-97a0-baa59f13b954-metrics-client-ca\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.829551 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.829515    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.829742 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.829639    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.830405 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.829911    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.830405 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.830181    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-thanos-querier-tls\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.830405 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.830196    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-grpc-tls\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.830405 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.830242    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a6fa390-9d6a-4590-97a0-baa59f13b954-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:58.833883 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:58.833858    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcz5s\" (UniqueName: \"kubernetes.io/projected/4a6fa390-9d6a-4590-97a0-baa59f13b954-kube-api-access-xcz5s\") pod \"thanos-querier-7c68d77877-fkzdg\" (UID: \"4a6fa390-9d6a-4590-97a0-baa59f13b954\") " pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:59.006403 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:59.006370    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:54:59.096345 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:54:59.096315    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tfrst"
Jul 16 04:55:00.015162 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.015125    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b6b48b445-m7brf"]
Jul 16 04:55:00.018771 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.018742    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.021343 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.021312    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 16 04:55:00.021506 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.021319    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 16 04:55:00.021506 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.021381    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pb7fm\""
Jul 16 04:55:00.022423 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.022287    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 16 04:55:00.022423 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.022325    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 16 04:55:00.022763 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.022511    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f4bl3q887qrn3\""
Jul 16 04:55:00.028755 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.028466    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b6b48b445-m7brf"]
Jul 16 04:55:00.138712 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.138671    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-secret-metrics-server-client-certs\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.138905 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.138724    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-secret-metrics-server-tls\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.138905 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.138756    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-metrics-server-audit-profiles\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.138905 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.138861    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-client-ca-bundle\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.139054 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.138917    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.139054 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.138957    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-audit-log\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.139154 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.139092    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr7k2\" (UniqueName: \"kubernetes.io/projected/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-kube-api-access-hr7k2\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.239894 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.239857    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr7k2\" (UniqueName: \"kubernetes.io/projected/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-kube-api-access-hr7k2\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.240113 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.239966    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-secret-metrics-server-client-certs\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.240113 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.240006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-secret-metrics-server-tls\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.240113 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.240034    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-metrics-server-audit-profiles\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.240113 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.240092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-client-ca-bundle\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.240347 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.240133    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.240347 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.240159    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-audit-log\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.241026 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.240994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-audit-log\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.241280 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.241250    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.241423 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.241394    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-metrics-server-audit-profiles\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.243087 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.243061    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-secret-metrics-server-client-certs\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.243418 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.243395    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-secret-metrics-server-tls\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.243967 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.243941    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-client-ca-bundle\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.248339 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.248308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr7k2\" (UniqueName: \"kubernetes.io/projected/c830977f-610a-4d9e-bf1e-a2d93fe4a54f-kube-api-access-hr7k2\") pod \"metrics-server-7b6b48b445-m7brf\" (UID: \"c830977f-610a-4d9e-bf1e-a2d93fe4a54f\") " pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.332403 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.332320    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:00.507866 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.507701    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b8d7cf447-fxmqb"]
Jul 16 04:55:00.509567 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.509524    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv"]
Jul 16 04:55:00.513272 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.513240    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv"
Jul 16 04:55:00.515869 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.515841    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 16 04:55:00.516180 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.516158    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zhq29\""
Jul 16 04:55:00.522231 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.522159    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv"]
Jul 16 04:55:00.543246 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.543208    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7412f889-1312-4254-ba2d-fbc59c89b468-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-sdlvv\" (UID: \"7412f889-1312-4254-ba2d-fbc59c89b468\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv"
Jul 16 04:55:00.543447 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.543308    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmqvd\" (UniqueName: \"kubernetes.io/projected/7412f889-1312-4254-ba2d-fbc59c89b468-kube-api-access-mmqvd\") pod \"monitoring-plugin-5f68df7f58-sdlvv\" (UID: \"7412f889-1312-4254-ba2d-fbc59c89b468\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv"
Jul 16 04:55:00.644719 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.644583    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7412f889-1312-4254-ba2d-fbc59c89b468-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-sdlvv\" (UID: \"7412f889-1312-4254-ba2d-fbc59c89b468\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv"
Jul 16 04:55:00.644899 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:55:00.644721    2565 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 16 04:55:00.644899 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.644729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmqvd\" (UniqueName: \"kubernetes.io/projected/7412f889-1312-4254-ba2d-fbc59c89b468-kube-api-access-mmqvd\") pod \"monitoring-plugin-5f68df7f58-sdlvv\" (UID: \"7412f889-1312-4254-ba2d-fbc59c89b468\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv"
Jul 16 04:55:00.644899 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:55:00.644803    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7412f889-1312-4254-ba2d-fbc59c89b468-monitoring-plugin-cert podName:7412f889-1312-4254-ba2d-fbc59c89b468 nodeName:}" failed. No retries permitted until 2026-07-16 04:55:01.144778303 +0000 UTC m=+304.560733374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/7412f889-1312-4254-ba2d-fbc59c89b468-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-sdlvv" (UID: "7412f889-1312-4254-ba2d-fbc59c89b468") : secret "monitoring-plugin-cert" not found
Jul 16 04:55:00.653710 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.653656    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmqvd\" (UniqueName: \"kubernetes.io/projected/7412f889-1312-4254-ba2d-fbc59c89b468-kube-api-access-mmqvd\") pod \"monitoring-plugin-5f68df7f58-sdlvv\" (UID: \"7412f889-1312-4254-ba2d-fbc59c89b468\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv"
Jul 16 04:55:00.941926 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.941827    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-df4686d48-qdq7s"]
Jul 16 04:55:00.946166 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.946121    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:00.948878 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.948849    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 16 04:55:00.949052 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.949023    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 16 04:55:00.949159 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.949136    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 16 04:55:00.949289 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.949240    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 16 04:55:00.949361 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.949248    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-hjz4v\""
Jul 16 04:55:00.949417 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.949376    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 16 04:55:00.954156 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.954128    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 16 04:55:00.959557 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:00.959516    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-df4686d48-qdq7s"]
Jul 16 04:55:01.049163 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.049125    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lfkb\" (UniqueName: \"kubernetes.io/projected/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-kube-api-access-6lfkb\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.049633 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.049191    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-secret-telemeter-client\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.049633 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.049253    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-serving-certs-ca-bundle\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.049633 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.049348    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-federate-client-tls\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.049633 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.049400    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-metrics-client-ca\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.049633 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.049425    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-telemeter-client-tls\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.049633 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.049455    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.049633 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.049480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-telemeter-trusted-ca-bundle\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.149999 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.149959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-federate-client-tls\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.150215 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.150041    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-metrics-client-ca\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.150215 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.150072    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-telemeter-client-tls\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.150215 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.150115    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.150215 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.150147    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-telemeter-trusted-ca-bundle\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.150215 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.150189    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lfkb\" (UniqueName: \"kubernetes.io/projected/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-kube-api-access-6lfkb\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.150477 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.150231    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-secret-telemeter-client\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.150477 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.150278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7412f889-1312-4254-ba2d-fbc59c89b468-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-sdlvv\" (UID: \"7412f889-1312-4254-ba2d-fbc59c89b468\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv"
Jul 16 04:55:01.150477 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.150296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-serving-certs-ca-bundle\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.151211 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.151178    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-serving-certs-ca-bundle\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.151391 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.150997    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-metrics-client-ca\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.151823 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.151794    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-telemeter-trusted-ca-bundle\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.153253 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.153228    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-telemeter-client-tls\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.153422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.153398    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-federate-client-tls\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.153839 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.153817    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.154056 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.154036    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7412f889-1312-4254-ba2d-fbc59c89b468-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-sdlvv\" (UID: \"7412f889-1312-4254-ba2d-fbc59c89b468\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv"
Jul 16 04:55:01.154520 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.154494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-secret-telemeter-client\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.160662 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.160632    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lfkb\" (UniqueName: \"kubernetes.io/projected/4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12-kube-api-access-6lfkb\") pod \"telemeter-client-df4686d48-qdq7s\" (UID: \"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12\") " pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.258637 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.258578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s"
Jul 16 04:55:01.426207 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:01.426167    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv"
Jul 16 04:55:03.037150 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:03.037104    2565 patch_prober.go:28] interesting pod/image-registry-776bb7cd46-j6xbd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 16 04:55:03.037569 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:03.037170    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd" podUID="40f2c277-95a8-4dc0-acda-1fd724cd9c0f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 04:55:03.136534 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:03.136491    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:55:03.278914 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:03.278870    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:55:05.490518 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:05.490484    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:55:05.491012 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:05.490554    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:55:05.496532 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:05.496504    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:55:06.049241 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.049139    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b6b48b445-m7brf"]
Jul 16 04:55:06.068976 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.068076    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:55:06.069397 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.069375    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 04:55:06.078924 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.076869    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c68d77877-fkzdg"]
Jul 16 04:55:06.085457 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:55:06.085385    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6fa390_9d6a_4590_97a0_baa59f13b954.slice/crio-7f4ad5639730eb630f8e5ac6bc386682e3906830ebb3da1b899917914f2bee4e WatchSource:0}: Error finding container 7f4ad5639730eb630f8e5ac6bc386682e3906830ebb3da1b899917914f2bee4e: Status 404 returned error can't find the container with id 7f4ad5639730eb630f8e5ac6bc386682e3906830ebb3da1b899917914f2bee4e
Jul 16 04:55:06.098394 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.098341    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv"]
Jul 16 04:55:06.102433 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:55:06.102394    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7412f889_1312_4254_ba2d_fbc59c89b468.slice/crio-a350541326bd95e29794aa1f0095f3c444eea8044880528e989203248d8e1abd WatchSource:0}: Error finding container a350541326bd95e29794aa1f0095f3c444eea8044880528e989203248d8e1abd: Status 404 returned error can't find the container with id a350541326bd95e29794aa1f0095f3c444eea8044880528e989203248d8e1abd
Jul 16 04:55:06.105908 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.105877    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-df4686d48-qdq7s"]
Jul 16 04:55:06.178313 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.178278    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-wbzhk" event={"ID":"00dab318-0248-4bf4-b18b-d439c871101d","Type":"ContainerStarted","Data":"118ad9a37f6645ea0bb24887bdb9230d944ca3a830700b2f9872ae97b85f3a64"}
Jul 16 04:55:06.178492 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.178457    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-wbzhk"
Jul 16 04:55:06.180128 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.180096    2565 patch_prober.go:28] interesting pod/downloads-68c9cbb497-wbzhk container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.19:8080/\": dial tcp 10.132.0.19:8080: connect: connection refused" start-of-body=
Jul 16 04:55:06.180286 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.180154    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-68c9cbb497-wbzhk" podUID="00dab318-0248-4bf4-b18b-d439c871101d" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.19:8080/\": dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 16 04:55:06.180802 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.180763    2565 generic.go:356] "Generic (PLEG): container finished" podID="1ce6ade3-c5ef-48dc-aec9-85277e8472fd" containerID="282dd5d22a07bf132e94a7399d70feec32d8f4acf99a2c0b8bacc1c46dfe8c81" exitCode=0
Jul 16 04:55:06.180926 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.180883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7rtjs" event={"ID":"1ce6ade3-c5ef-48dc-aec9-85277e8472fd","Type":"ContainerDied","Data":"282dd5d22a07bf132e94a7399d70feec32d8f4acf99a2c0b8bacc1c46dfe8c81"}
Jul 16 04:55:06.183073 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.183039    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg" event={"ID":"4a6fa390-9d6a-4590-97a0-baa59f13b954","Type":"ContainerStarted","Data":"7f4ad5639730eb630f8e5ac6bc386682e3906830ebb3da1b899917914f2bee4e"}
Jul 16 04:55:06.185698 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.185673    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s" event={"ID":"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12","Type":"ContainerStarted","Data":"36eebfa9619b3cd1143fac15c856517a6d212ac828bbfa8238f349becf8112a8"}
Jul 16 04:55:06.187324 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.187289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf" event={"ID":"c830977f-610a-4d9e-bf1e-a2d93fe4a54f","Type":"ContainerStarted","Data":"debac60cbf0d69c4e3d6e74e7de319bf68290b2040e6aa19482ff77397cfbf94"}
Jul 16 04:55:06.188775 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.188749    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv" event={"ID":"7412f889-1312-4254-ba2d-fbc59c89b468","Type":"ContainerStarted","Data":"a350541326bd95e29794aa1f0095f3c444eea8044880528e989203248d8e1abd"}
Jul 16 04:55:06.195342 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.195295    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-wbzhk" podStartSLOduration=3.590156647 podStartE2EDuration="22.195274912s" totalImagesPullingTime="18.605118265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:55:06.193800148 +0000 UTC m=+309.609755225" watchObservedRunningTime="2026-07-16 04:55:06.195274912 +0000 UTC m=+309.611229984"
Jul 16 04:55:06.198795 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:06.197710    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:55:07.213412 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:07.213294    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd" event={"ID":"ea86cd3f-b598-443a-8d70-cb2fc310aecd","Type":"ContainerStarted","Data":"976b9e076706247693e751d939918015d93ce9227bed9940e5e11eb8fa6a8154"}
Jul 16 04:55:07.218409 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:07.218375    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7rtjs" event={"ID":"1ce6ade3-c5ef-48dc-aec9-85277e8472fd","Type":"ContainerStarted","Data":"cfe6a862bb83be213d1fae7feb6192cf43597b96656596426e4c1caa944410fa"}
Jul 16 04:55:07.218739 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:07.218433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7rtjs" event={"ID":"1ce6ade3-c5ef-48dc-aec9-85277e8472fd","Type":"ContainerStarted","Data":"d236a17432abddbce585874194ec5bf84f99ff0c7d6e3e7042ae7c777c13c999"}
Jul 16 04:55:07.227164 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:07.227131    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-wbzhk"
Jul 16 04:55:07.270750 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:07.270646    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7rtjs" podStartSLOduration=3.580069215 podStartE2EDuration="12.270620538s" totalImagesPullingTime="8.574770379s" totalInitContainerRuntime="115.780944ms" isStatefulPod=true podCreationTimestamp="2026-07-16 04:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:55:07.250623862 +0000 UTC m=+310.666578936" watchObservedRunningTime="2026-07-16 04:55:07.270620538 +0000 UTC m=+310.686575614"
Jul 16 04:55:07.757669 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:07.757259    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-776bb7cd46-j6xbd"]
Jul 16 04:55:08.230986 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:08.230187    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd" event={"ID":"ea86cd3f-b598-443a-8d70-cb2fc310aecd","Type":"ContainerStarted","Data":"c5b19c1d55e43f8a34692a286c5d57c4710a67753250ebb49f849c7ef55d6718"}
Jul 16 04:55:08.230986 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:08.230258    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd" event={"ID":"ea86cd3f-b598-443a-8d70-cb2fc310aecd","Type":"ContainerStarted","Data":"25ae20d60e6b535ceb52790e57f920bec189c5d7e39703d709275f5b787c02a9"}
Jul 16 04:55:08.250494 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:08.249141    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-9zsxd" podStartSLOduration=4.064498948 podStartE2EDuration="13.249115711s" totalImagesPullingTime="9.184616763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:55:08.247150451 +0000 UTC m=+311.663105524" watchObservedRunningTime="2026-07-16 04:55:08.249115711 +0000 UTC m=+311.665070786"
Jul 16 04:55:11.862461 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:11.862423    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5679894dd6-4gm5p"]
Jul 16 04:55:12.261115 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:12.261058    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg" event={"ID":"4a6fa390-9d6a-4590-97a0-baa59f13b954","Type":"ContainerStarted","Data":"da4e5d45be82c5f061b472c82a23075447eb551ac208b8323b07d49d210fa238"}
Jul 16 04:55:12.261295 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:12.261154    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg" event={"ID":"4a6fa390-9d6a-4590-97a0-baa59f13b954","Type":"ContainerStarted","Data":"dc20a7146a68299aa9a1535a1d5a9417b0b0cb35e446f5cdd7027d3b6d7fd836"}
Jul 16 04:55:12.264423 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:12.263760    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s" event={"ID":"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12","Type":"ContainerStarted","Data":"87542827b976b46719461379f4b8141096b0245a4d4e237bb391fcc074df9a2a"}
Jul 16 04:55:12.268233 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:12.265979    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf" event={"ID":"c830977f-610a-4d9e-bf1e-a2d93fe4a54f","Type":"ContainerStarted","Data":"b09fb7d4774577dcaebeafe5ea29d0eaa774f64dd004e4d609d4284053781ceb"}
Jul 16 04:55:12.271205 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:12.270570    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv" event={"ID":"7412f889-1312-4254-ba2d-fbc59c89b468","Type":"ContainerStarted","Data":"5f4ad7aed68bea78ddf7e698ff727de5b56838f0ff0e00c5a6cacb712853621c"}
Jul 16 04:55:12.271205 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:12.271145    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv"
Jul 16 04:55:12.278771 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:12.278730    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv"
Jul 16 04:55:12.284570 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:12.284518    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf" podStartSLOduration=7.470265474 podStartE2EDuration="13.284499275s" totalImagesPullingTime="5.814233801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:54:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:55:12.28252765 +0000 UTC m=+315.698482761" watchObservedRunningTime="2026-07-16 04:55:12.284499275 +0000 UTC m=+315.700454347"
Jul 16 04:55:12.299910 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:12.299858    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-sdlvv" podStartSLOduration=6.5263124900000005 podStartE2EDuration="12.299838922s" totalImagesPullingTime="5.773526432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:55:12.298877987 +0000 UTC m=+315.714833061" watchObservedRunningTime="2026-07-16 04:55:12.299838922 +0000 UTC m=+315.715794036"
Jul 16 04:55:13.278763 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:13.278716    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg" event={"ID":"4a6fa390-9d6a-4590-97a0-baa59f13b954","Type":"ContainerStarted","Data":"459dcfe917a20f0794f3927a9c6cf7f15a1a52a6a23d3bbd14c52da11c9dca98"}
Jul 16 04:55:14.287782 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:14.287743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg" event={"ID":"4a6fa390-9d6a-4590-97a0-baa59f13b954","Type":"ContainerStarted","Data":"ec39b3d94a87282770f65aaf2d92eaaceb8ed2d3902ddc3f6ca2269b5c9cd368"}
Jul 16 04:55:14.288252 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:14.287836    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg" event={"ID":"4a6fa390-9d6a-4590-97a0-baa59f13b954","Type":"ContainerStarted","Data":"58924e0161ef88079608d99307897c640703ae6d6edc9a2c544fbb86b6b3baee"}
Jul 16 04:55:14.290906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:14.290803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s" event={"ID":"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12","Type":"ContainerStarted","Data":"7b6835e5b56367d07dc72724f41f6ac3dd29fa1177d8c0c5d038251ebadb2a81"}
Jul 16 04:55:14.290906 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:14.290867    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s" event={"ID":"4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12","Type":"ContainerStarted","Data":"02d4108b9e0359d1ec27bf7eef34d45a764e18a8f2e5420e3ab69b20f17995de"}
Jul 16 04:55:14.313813 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:14.313756    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-df4686d48-qdq7s" podStartSLOduration=6.715437212 podStartE2EDuration="14.313738144s" totalImagesPullingTime="7.598300932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:55:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:55:14.312615755 +0000 UTC m=+317.728570819" watchObservedRunningTime="2026-07-16 04:55:14.313738144 +0000 UTC m=+317.729693216"
Jul 16 04:55:15.298210 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:15.298165    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg" event={"ID":"4a6fa390-9d6a-4590-97a0-baa59f13b954","Type":"ContainerStarted","Data":"215b54e24e39ee07a75e182aff27403168fbe72e5ad8bb601e54923d68fd2480"}
Jul 16 04:55:15.319715 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:15.319660    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg" podStartSLOduration=9.959190106 podStartE2EDuration="17.319641243s" totalImagesPullingTime="7.360451137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:54:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:55:15.317744617 +0000 UTC m=+318.733699692" watchObservedRunningTime="2026-07-16 04:55:15.319641243 +0000 UTC m=+318.735596356"
Jul 16 04:55:16.310071 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:16.310036    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:55:17.321208 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:17.321177    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c68d77877-fkzdg"
Jul 16 04:55:20.332564 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:20.332527    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:20.333036 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:20.332621    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:25.538562 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:25.538491    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b8d7cf447-fxmqb" podUID="6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43" containerName="console" containerID="cri-o://8cfe76e172bb39b05faefb48914cd0e6eeffa9ca334ded6fc30783a42bdc3a62" gracePeriod=15
Jul 16 04:55:25.825892 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:25.825809    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:55:25.997731 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:25.997690    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-oauth-config\") pod \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") "
Jul 16 04:55:25.997888 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:25.997744    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-oauth-serving-cert\") pod \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") "
Jul 16 04:55:25.997888 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:25.997779    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-config\") pod \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") "
Jul 16 04:55:25.998002 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:25.997899    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-service-ca\") pod \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") "
Jul 16 04:55:25.998002 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:25.997965    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-serving-cert\") pod \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") "
Jul 16 04:55:25.998116 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:25.998003    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w26s4\" (UniqueName: \"kubernetes.io/projected/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-kube-api-access-w26s4\") pod \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\" (UID: \"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43\") "
Jul 16 04:55:25.998172 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:25.998122    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-config" (OuterVolumeSpecName: "console-config") pod "6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43" (UID: "6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 04:55:25.998252 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:25.998216    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43" (UID: "6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 04:55:25.998311 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:25.998261    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-service-ca" (OuterVolumeSpecName: "service-ca") pod "6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43" (UID: "6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 04:55:25.998388 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:25.998368    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-oauth-serving-cert\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:25.998443 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:25.998397    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-config\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:25.998443 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:25.998411    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-service-ca\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:26.000430 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:26.000401    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-kube-api-access-w26s4" (OuterVolumeSpecName: "kube-api-access-w26s4") pod "6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43" (UID: "6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43"). InnerVolumeSpecName "kube-api-access-w26s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 04:55:26.000545 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:26.000436    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43" (UID: "6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 04:55:26.000545 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:26.000453    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43" (UID: "6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 04:55:26.099945 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:26.099864    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-serving-cert\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:26.099945 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:26.099892    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w26s4\" (UniqueName: \"kubernetes.io/projected/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-kube-api-access-w26s4\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:26.099945 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:26.099902    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43-console-oauth-config\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:26.362114 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:26.362029    2565 generic.go:356] "Generic (PLEG): container finished" podID="6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43" containerID="8cfe76e172bb39b05faefb48914cd0e6eeffa9ca334ded6fc30783a42bdc3a62" exitCode=0
Jul 16 04:55:26.362114 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:26.362070    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8d7cf447-fxmqb"
Jul 16 04:55:26.362301 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:26.362074    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8d7cf447-fxmqb" event={"ID":"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43","Type":"ContainerDied","Data":"8cfe76e172bb39b05faefb48914cd0e6eeffa9ca334ded6fc30783a42bdc3a62"}
Jul 16 04:55:26.362301 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:26.362186    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8d7cf447-fxmqb" event={"ID":"6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43","Type":"ContainerDied","Data":"3d63e720c038cdec9d0123067ccfaa1c8cccd7da6368d1a1539aa9efd10548f6"}
Jul 16 04:55:26.362301 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:26.362214    2565 scope.go:117] "RemoveContainer" containerID="8cfe76e172bb39b05faefb48914cd0e6eeffa9ca334ded6fc30783a42bdc3a62"
Jul 16 04:55:26.372262 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:26.372241    2565 scope.go:117] "RemoveContainer" containerID="8cfe76e172bb39b05faefb48914cd0e6eeffa9ca334ded6fc30783a42bdc3a62"
Jul 16 04:55:26.372565 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:55:26.372543    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cfe76e172bb39b05faefb48914cd0e6eeffa9ca334ded6fc30783a42bdc3a62\": container with ID starting with 8cfe76e172bb39b05faefb48914cd0e6eeffa9ca334ded6fc30783a42bdc3a62 not found: ID does not exist" containerID="8cfe76e172bb39b05faefb48914cd0e6eeffa9ca334ded6fc30783a42bdc3a62"
Jul 16 04:55:26.372636 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:26.372574    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cfe76e172bb39b05faefb48914cd0e6eeffa9ca334ded6fc30783a42bdc3a62"} err="failed to get container status \"8cfe76e172bb39b05faefb48914cd0e6eeffa9ca334ded6fc30783a42bdc3a62\": rpc error: code = NotFound desc = could not find container \"8cfe76e172bb39b05faefb48914cd0e6eeffa9ca334ded6fc30783a42bdc3a62\": container with ID starting with 8cfe76e172bb39b05faefb48914cd0e6eeffa9ca334ded6fc30783a42bdc3a62 not found: ID does not exist"
Jul 16 04:55:26.383320 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:26.383280    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b8d7cf447-fxmqb"]
Jul 16 04:55:26.385527 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:26.385499    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b8d7cf447-fxmqb"]
Jul 16 04:55:27.141169 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:27.141131    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43" path="/var/lib/kubelet/pods/6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43/volumes"
Jul 16 04:55:32.792093 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:32.792017    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd" podUID="40f2c277-95a8-4dc0-acda-1fd724cd9c0f" containerName="registry" containerID="cri-o://7b9aa874980aa2f8d6af448d0705c1de3e95c78a649c6ac0a723f92a168fc10a" gracePeriod=30
Jul 16 04:55:33.104509 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.104468    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:55:33.264751 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.264711    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-ca-trust-extracted\") pod \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") "
Jul 16 04:55:33.264939 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.264772    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-bound-sa-token\") pod \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") "
Jul 16 04:55:33.264939 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.264798    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls\") pod \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") "
Jul 16 04:55:33.264939 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.264826    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-installation-pull-secrets\") pod \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") "
Jul 16 04:55:33.264939 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.264863    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-image-registry-private-configuration\") pod \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") "
Jul 16 04:55:33.264939 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.264900    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-certificates\") pod \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") "
Jul 16 04:55:33.265188 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.264951    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-trusted-ca\") pod \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") "
Jul 16 04:55:33.265188 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.265001    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smptp\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-kube-api-access-smptp\") pod \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\" (UID: \"40f2c277-95a8-4dc0-acda-1fd724cd9c0f\") "
Jul 16 04:55:33.265538 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.265457    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "40f2c277-95a8-4dc0-acda-1fd724cd9c0f" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 04:55:33.265538 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.265493    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "40f2c277-95a8-4dc0-acda-1fd724cd9c0f" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 04:55:33.267750 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.267712    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "40f2c277-95a8-4dc0-acda-1fd724cd9c0f" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 04:55:33.267986 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.267950    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "40f2c277-95a8-4dc0-acda-1fd724cd9c0f" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 04:55:33.267986 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.267968    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-kube-api-access-smptp" (OuterVolumeSpecName: "kube-api-access-smptp") pod "40f2c277-95a8-4dc0-acda-1fd724cd9c0f" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f"). InnerVolumeSpecName "kube-api-access-smptp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 04:55:33.267986 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.267977    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "40f2c277-95a8-4dc0-acda-1fd724cd9c0f" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 04:55:33.268187 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.268052    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "40f2c277-95a8-4dc0-acda-1fd724cd9c0f" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 04:55:33.273492 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.273456    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "40f2c277-95a8-4dc0-acda-1fd724cd9c0f" (UID: "40f2c277-95a8-4dc0-acda-1fd724cd9c0f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 04:55:33.365815 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.365721    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-certificates\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:33.365815 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.365757    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-trusted-ca\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:33.365815 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.365766    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smptp\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-kube-api-access-smptp\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:33.365815 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.365775    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-ca-trust-extracted\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:33.365815 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.365783    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-bound-sa-token\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:33.365815 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.365791    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-registry-tls\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:33.365815 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.365799    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-installation-pull-secrets\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:33.365815 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.365807    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40f2c277-95a8-4dc0-acda-1fd724cd9c0f-image-registry-private-configuration\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:33.400065 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.400033    2565 generic.go:356] "Generic (PLEG): container finished" podID="95bf5f5b-d8bb-4638-895a-015e35aea09a" containerID="ce189da53d6814803eedf53dc011d8df64fcd45ebbc697b2a9226671157fbbd1" exitCode=0
Jul 16 04:55:33.400235 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.400111    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8" event={"ID":"95bf5f5b-d8bb-4638-895a-015e35aea09a","Type":"ContainerDied","Data":"ce189da53d6814803eedf53dc011d8df64fcd45ebbc697b2a9226671157fbbd1"}
Jul 16 04:55:33.400512 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.400493    2565 scope.go:117] "RemoveContainer" containerID="ce189da53d6814803eedf53dc011d8df64fcd45ebbc697b2a9226671157fbbd1"
Jul 16 04:55:33.401431 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.401407    2565 generic.go:356] "Generic (PLEG): container finished" podID="40f2c277-95a8-4dc0-acda-1fd724cd9c0f" containerID="7b9aa874980aa2f8d6af448d0705c1de3e95c78a649c6ac0a723f92a168fc10a" exitCode=0
Jul 16 04:55:33.401550 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.401470    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd"
Jul 16 04:55:33.401550 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.401493    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd" event={"ID":"40f2c277-95a8-4dc0-acda-1fd724cd9c0f","Type":"ContainerDied","Data":"7b9aa874980aa2f8d6af448d0705c1de3e95c78a649c6ac0a723f92a168fc10a"}
Jul 16 04:55:33.401550 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.401546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-776bb7cd46-j6xbd" event={"ID":"40f2c277-95a8-4dc0-acda-1fd724cd9c0f","Type":"ContainerDied","Data":"a5a85e8276dc1d75cf510842d309a11f1c3b8efb63991bfcb620ab7d2691d50e"}
Jul 16 04:55:33.401730 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.401572    2565 scope.go:117] "RemoveContainer" containerID="7b9aa874980aa2f8d6af448d0705c1de3e95c78a649c6ac0a723f92a168fc10a"
Jul 16 04:55:33.412986 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.412960    2565 scope.go:117] "RemoveContainer" containerID="7b9aa874980aa2f8d6af448d0705c1de3e95c78a649c6ac0a723f92a168fc10a"
Jul 16 04:55:33.413325 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:55:33.413294    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9aa874980aa2f8d6af448d0705c1de3e95c78a649c6ac0a723f92a168fc10a\": container with ID starting with 7b9aa874980aa2f8d6af448d0705c1de3e95c78a649c6ac0a723f92a168fc10a not found: ID does not exist" containerID="7b9aa874980aa2f8d6af448d0705c1de3e95c78a649c6ac0a723f92a168fc10a"
Jul 16 04:55:33.413417 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.413325    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9aa874980aa2f8d6af448d0705c1de3e95c78a649c6ac0a723f92a168fc10a"} err="failed to get container status \"7b9aa874980aa2f8d6af448d0705c1de3e95c78a649c6ac0a723f92a168fc10a\": rpc error: code = NotFound desc = could not find container \"7b9aa874980aa2f8d6af448d0705c1de3e95c78a649c6ac0a723f92a168fc10a\": container with ID starting with 7b9aa874980aa2f8d6af448d0705c1de3e95c78a649c6ac0a723f92a168fc10a not found: ID does not exist"
Jul 16 04:55:33.432850 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.432815    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-776bb7cd46-j6xbd"]
Jul 16 04:55:33.437935 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:33.437905    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-776bb7cd46-j6xbd"]
Jul 16 04:55:34.407523 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:34.407483    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2xgh8" event={"ID":"95bf5f5b-d8bb-4638-895a-015e35aea09a","Type":"ContainerStarted","Data":"c41b54eafabfb464262946edd1060c901a66caece2dbc88319a236288228c99b"}
Jul 16 04:55:35.141002 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:35.140969    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40f2c277-95a8-4dc0-acda-1fd724cd9c0f" path="/var/lib/kubelet/pods/40f2c277-95a8-4dc0-acda-1fd724cd9c0f/volumes"
Jul 16 04:55:36.891927 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:36.891878    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5679894dd6-4gm5p" podUID="1901fb00-4f59-47e2-ae36-d51a76acb59c" containerName="console" containerID="cri-o://5d078b011d316a061e31e1f93d5477c6778faa55b325050d950fe607e2607a4e" gracePeriod=15
Jul 16 04:55:37.182793 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.182750    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:55:37.197101 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.197068    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-serving-cert\") pod \"1901fb00-4f59-47e2-ae36-d51a76acb59c\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") "
Jul 16 04:55:37.197264 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.197114    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-config\") pod \"1901fb00-4f59-47e2-ae36-d51a76acb59c\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") "
Jul 16 04:55:37.197264 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.197134    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-trusted-ca-bundle\") pod \"1901fb00-4f59-47e2-ae36-d51a76acb59c\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") "
Jul 16 04:55:37.197264 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.197187    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rllz9\" (UniqueName: \"kubernetes.io/projected/1901fb00-4f59-47e2-ae36-d51a76acb59c-kube-api-access-rllz9\") pod \"1901fb00-4f59-47e2-ae36-d51a76acb59c\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") "
Jul 16 04:55:37.197264 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.197229    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-oauth-config\") pod \"1901fb00-4f59-47e2-ae36-d51a76acb59c\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") "
Jul 16 04:55:37.197264 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.197258    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-oauth-serving-cert\") pod \"1901fb00-4f59-47e2-ae36-d51a76acb59c\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") "
Jul 16 04:55:37.197546 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.197298    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-service-ca\") pod \"1901fb00-4f59-47e2-ae36-d51a76acb59c\" (UID: \"1901fb00-4f59-47e2-ae36-d51a76acb59c\") "
Jul 16 04:55:37.197787 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.197753    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1901fb00-4f59-47e2-ae36-d51a76acb59c" (UID: "1901fb00-4f59-47e2-ae36-d51a76acb59c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 04:55:37.198722 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.198685    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-service-ca" (OuterVolumeSpecName: "service-ca") pod "1901fb00-4f59-47e2-ae36-d51a76acb59c" (UID: "1901fb00-4f59-47e2-ae36-d51a76acb59c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 04:55:37.198995 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.198971    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-config" (OuterVolumeSpecName: "console-config") pod "1901fb00-4f59-47e2-ae36-d51a76acb59c" (UID: "1901fb00-4f59-47e2-ae36-d51a76acb59c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 04:55:37.199077 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.199064    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1901fb00-4f59-47e2-ae36-d51a76acb59c" (UID: "1901fb00-4f59-47e2-ae36-d51a76acb59c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 04:55:37.200411 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.200368    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1901fb00-4f59-47e2-ae36-d51a76acb59c-kube-api-access-rllz9" (OuterVolumeSpecName: "kube-api-access-rllz9") pod "1901fb00-4f59-47e2-ae36-d51a76acb59c" (UID: "1901fb00-4f59-47e2-ae36-d51a76acb59c"). InnerVolumeSpecName "kube-api-access-rllz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 04:55:37.200756 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.200730    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1901fb00-4f59-47e2-ae36-d51a76acb59c" (UID: "1901fb00-4f59-47e2-ae36-d51a76acb59c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 04:55:37.200858 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.200781    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1901fb00-4f59-47e2-ae36-d51a76acb59c" (UID: "1901fb00-4f59-47e2-ae36-d51a76acb59c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 04:55:37.298559 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.298518    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-serving-cert\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:37.298559 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.298551    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-config\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:37.298559 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.298562    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-trusted-ca-bundle\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:37.298559 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.298571    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rllz9\" (UniqueName: \"kubernetes.io/projected/1901fb00-4f59-47e2-ae36-d51a76acb59c-kube-api-access-rllz9\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:37.298851 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.298580    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1901fb00-4f59-47e2-ae36-d51a76acb59c-console-oauth-config\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:37.298851 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.298604    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-oauth-serving-cert\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:37.298851 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.298613    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1901fb00-4f59-47e2-ae36-d51a76acb59c-service-ca\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:55:37.424762 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.424520    2565 generic.go:356] "Generic (PLEG): container finished" podID="1901fb00-4f59-47e2-ae36-d51a76acb59c" containerID="5d078b011d316a061e31e1f93d5477c6778faa55b325050d950fe607e2607a4e" exitCode=0
Jul 16 04:55:37.424762 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.424558    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5679894dd6-4gm5p"
Jul 16 04:55:37.424950 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.424575    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5679894dd6-4gm5p" event={"ID":"1901fb00-4f59-47e2-ae36-d51a76acb59c","Type":"ContainerDied","Data":"5d078b011d316a061e31e1f93d5477c6778faa55b325050d950fe607e2607a4e"}
Jul 16 04:55:37.424950 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.424873    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5679894dd6-4gm5p" event={"ID":"1901fb00-4f59-47e2-ae36-d51a76acb59c","Type":"ContainerDied","Data":"5694c7385e5a401c5436ce97dd472075fcc6f7461375be006edbdbbc8b42ecfa"}
Jul 16 04:55:37.424950 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.424898    2565 scope.go:117] "RemoveContainer" containerID="5d078b011d316a061e31e1f93d5477c6778faa55b325050d950fe607e2607a4e"
Jul 16 04:55:37.440297 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.440275    2565 scope.go:117] "RemoveContainer" containerID="5d078b011d316a061e31e1f93d5477c6778faa55b325050d950fe607e2607a4e"
Jul 16 04:55:37.440653 ip-10-0-138-199 kubenswrapper[2565]: E0716 04:55:37.440632    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d078b011d316a061e31e1f93d5477c6778faa55b325050d950fe607e2607a4e\": container with ID starting with 5d078b011d316a061e31e1f93d5477c6778faa55b325050d950fe607e2607a4e not found: ID does not exist" containerID="5d078b011d316a061e31e1f93d5477c6778faa55b325050d950fe607e2607a4e"
Jul 16 04:55:37.440726 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.440669    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d078b011d316a061e31e1f93d5477c6778faa55b325050d950fe607e2607a4e"} err="failed to get container status \"5d078b011d316a061e31e1f93d5477c6778faa55b325050d950fe607e2607a4e\": rpc error: code = NotFound desc = could not find container \"5d078b011d316a061e31e1f93d5477c6778faa55b325050d950fe607e2607a4e\": container with ID starting with 5d078b011d316a061e31e1f93d5477c6778faa55b325050d950fe607e2607a4e not found: ID does not exist"
Jul 16 04:55:37.445446 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.445415    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5679894dd6-4gm5p"]
Jul 16 04:55:37.449124 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:37.449094    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5679894dd6-4gm5p"]
Jul 16 04:55:39.140554 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:39.140516    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1901fb00-4f59-47e2-ae36-d51a76acb59c" path="/var/lib/kubelet/pods/1901fb00-4f59-47e2-ae36-d51a76acb59c/volumes"
Jul 16 04:55:40.338953 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:40.338919    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:40.342907 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:40.342884    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b6b48b445-m7brf"
Jul 16 04:55:53.494139 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:53.494103    2565 generic.go:356] "Generic (PLEG): container finished" podID="22b39ae9-d2af-4874-9225-bd9a7a7f6bbf" containerID="6ad908f8f4c262ed356b9c2b958a52d3d9a69f18a487b5f4b0a21637a85e9394" exitCode=0
Jul 16 04:55:53.494539 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:53.494149    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs" event={"ID":"22b39ae9-d2af-4874-9225-bd9a7a7f6bbf","Type":"ContainerDied","Data":"6ad908f8f4c262ed356b9c2b958a52d3d9a69f18a487b5f4b0a21637a85e9394"}
Jul 16 04:55:53.494539 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:53.494445    2565 scope.go:117] "RemoveContainer" containerID="6ad908f8f4c262ed356b9c2b958a52d3d9a69f18a487b5f4b0a21637a85e9394"
Jul 16 04:55:54.500748 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:55:54.500712    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-tcnzs" event={"ID":"22b39ae9-d2af-4874-9225-bd9a7a7f6bbf","Type":"ContainerStarted","Data":"f3069e47646bb4feb93ca135252aa74905539ed668b39409b8cfb3cbd3d7403c"}
Jul 16 04:56:11.094845 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:11.094809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs\") pod \"network-metrics-daemon-ldxlj\" (UID: \"c028e133-a290-4758-998c-24e158389825\") " pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:56:11.097219 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:11.097199    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028e133-a290-4758-998c-24e158389825-metrics-certs\") pod \"network-metrics-daemon-ldxlj\" (UID: \"c028e133-a290-4758-998c-24e158389825\") " pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:56:11.239823 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:11.239788    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vpdjw\""
Jul 16 04:56:11.247746 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:11.247717    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxlj"
Jul 16 04:56:11.393222 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:11.393194    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ldxlj"]
Jul 16 04:56:11.395648 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:56:11.395614    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc028e133_a290_4758_998c_24e158389825.slice/crio-a81e3de53f140248d50e05972069948172c071a1a3b965071ffd6c2e325ef85b WatchSource:0}: Error finding container a81e3de53f140248d50e05972069948172c071a1a3b965071ffd6c2e325ef85b: Status 404 returned error can't find the container with id a81e3de53f140248d50e05972069948172c071a1a3b965071ffd6c2e325ef85b
Jul 16 04:56:11.572075 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:11.572038    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldxlj" event={"ID":"c028e133-a290-4758-998c-24e158389825","Type":"ContainerStarted","Data":"a81e3de53f140248d50e05972069948172c071a1a3b965071ffd6c2e325ef85b"}
Jul 16 04:56:12.578058 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:12.578019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldxlj" event={"ID":"c028e133-a290-4758-998c-24e158389825","Type":"ContainerStarted","Data":"dc04f05568a196772fc23ff8d4a96a6fc6e18bb3f529dc143c230e856ba327f6"}
Jul 16 04:56:12.578457 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:12.578109    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldxlj" event={"ID":"c028e133-a290-4758-998c-24e158389825","Type":"ContainerStarted","Data":"9db4166506d83cbdf88ce82e4d2e9f6cea6e7081d1c25654b06cb0ca2404b674"}
Jul 16 04:56:12.594030 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:12.593963    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ldxlj" podStartSLOduration=373.804059624 podStartE2EDuration="6m14.593945714s" totalImagesPullingTime="789.88609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:56:12.591936886 +0000 UTC m=+376.007891959" watchObservedRunningTime="2026-07-16 04:56:12.593945714 +0000 UTC m=+376.009900787"
Jul 16 04:56:15.571278 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.571234    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-655d596b95-w4j6j"]
Jul 16 04:56:15.571787 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.571726    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43" containerName="console"
Jul 16 04:56:15.571787 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.571746    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43" containerName="console"
Jul 16 04:56:15.571787 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.571761    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40f2c277-95a8-4dc0-acda-1fd724cd9c0f" containerName="registry"
Jul 16 04:56:15.571787 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.571769    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="40f2c277-95a8-4dc0-acda-1fd724cd9c0f" containerName="registry"
Jul 16 04:56:15.571787 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.571778    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1901fb00-4f59-47e2-ae36-d51a76acb59c" containerName="console"
Jul 16 04:56:15.571787 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.571786    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1901fb00-4f59-47e2-ae36-d51a76acb59c" containerName="console"
Jul 16 04:56:15.572075 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.571878    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1901fb00-4f59-47e2-ae36-d51a76acb59c" containerName="console"
Jul 16 04:56:15.572075 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.571927    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="40f2c277-95a8-4dc0-acda-1fd724cd9c0f" containerName="registry"
Jul 16 04:56:15.572075 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.571937    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afa39ff-fec3-4bcc-b1a3-c2cd92df7e43" containerName="console"
Jul 16 04:56:15.573874 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.573852    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.577493 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.577120    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 16 04:56:15.577493 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.577162    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 16 04:56:15.577493 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.577205    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 16 04:56:15.577493 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.577401    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4bwwt\""
Jul 16 04:56:15.577827 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.577560    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 16 04:56:15.577827 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.577629    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 16 04:56:15.582731 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.582704    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 16 04:56:15.583781 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.583755    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-655d596b95-w4j6j"]
Jul 16 04:56:15.738802 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.738764    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-service-ca\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.738976 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.738812    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-config\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.738976 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.738839    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-oauth-serving-cert\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.738976 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.738872    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wwq8\" (UniqueName: \"kubernetes.io/projected/cad72dda-df2f-4541-91f3-573a5b98dbf7-kube-api-access-6wwq8\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.738976 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.738898    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-serving-cert\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.738976 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.738920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-oauth-config\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.739139 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.739010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-trusted-ca-bundle\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.839674 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.839556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-trusted-ca-bundle\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.839674 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.839634    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-service-ca\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.839914 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.839677    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-config\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.839914 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.839704    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-oauth-serving-cert\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.839914 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.839733    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wwq8\" (UniqueName: \"kubernetes.io/projected/cad72dda-df2f-4541-91f3-573a5b98dbf7-kube-api-access-6wwq8\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.839914 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.839760    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-serving-cert\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.839914 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.839794    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-oauth-config\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.840510 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.840439    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-service-ca\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.840510 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.840514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-config\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.840510 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.840514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-oauth-serving-cert\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.840510 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.840554    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-trusted-ca-bundle\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.842306 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.842276    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-oauth-config\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.842400 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.842371    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-serving-cert\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.848408 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.848382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wwq8\" (UniqueName: \"kubernetes.io/projected/cad72dda-df2f-4541-91f3-573a5b98dbf7-kube-api-access-6wwq8\") pod \"console-655d596b95-w4j6j\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") " pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:15.886156 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:15.886115    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:16.039499 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:16.039463    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-655d596b95-w4j6j"]
Jul 16 04:56:16.041912 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:56:16.041870    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad72dda_df2f_4541_91f3_573a5b98dbf7.slice/crio-f92d3862e4f22d8425cb8251e710ed21a96ec7347ee5bcf03fa4d2d6e27dfac7 WatchSource:0}: Error finding container f92d3862e4f22d8425cb8251e710ed21a96ec7347ee5bcf03fa4d2d6e27dfac7: Status 404 returned error can't find the container with id f92d3862e4f22d8425cb8251e710ed21a96ec7347ee5bcf03fa4d2d6e27dfac7
Jul 16 04:56:16.596896 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:16.596860    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655d596b95-w4j6j" event={"ID":"cad72dda-df2f-4541-91f3-573a5b98dbf7","Type":"ContainerStarted","Data":"105fa0cc8387be3b51cf45dbd8f1fcd223ff085f2fed40b57391f610ad9c9332"}
Jul 16 04:56:16.597462 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:16.596917    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655d596b95-w4j6j" event={"ID":"cad72dda-df2f-4541-91f3-573a5b98dbf7","Type":"ContainerStarted","Data":"f92d3862e4f22d8425cb8251e710ed21a96ec7347ee5bcf03fa4d2d6e27dfac7"}
Jul 16 04:56:16.615468 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:16.615408    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-655d596b95-w4j6j" podStartSLOduration=1.6153921869999999 podStartE2EDuration="1.615392187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:56:16.613419638 +0000 UTC m=+380.029374712" watchObservedRunningTime="2026-07-16 04:56:16.615392187 +0000 UTC m=+380.031347315"
Jul 16 04:56:25.886821 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:25.886773    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:25.887300 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:25.886841    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:25.891883 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:25.891856    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:56:26.644234 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:56:26.644207    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 04:57:40.300894 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.300845    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8"]
Jul 16 04:57:40.303618 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.303567    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8"
Jul 16 04:57:40.306163 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.306135    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 04:57:40.307068 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.307050    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 04:57:40.307126 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.307050    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mz6kf\""
Jul 16 04:57:40.313621 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.313574    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8"]
Jul 16 04:57:40.344818 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.344784    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6rh6\" (UniqueName: \"kubernetes.io/projected/c3ec9c16-6579-4bfe-be14-0405d5aa956a-kube-api-access-t6rh6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8\" (UID: \"c3ec9c16-6579-4bfe-be14-0405d5aa956a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8"
Jul 16 04:57:40.344988 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.344860    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3ec9c16-6579-4bfe-be14-0405d5aa956a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8\" (UID: \"c3ec9c16-6579-4bfe-be14-0405d5aa956a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8"
Jul 16 04:57:40.344988 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.344897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3ec9c16-6579-4bfe-be14-0405d5aa956a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8\" (UID: \"c3ec9c16-6579-4bfe-be14-0405d5aa956a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8"
Jul 16 04:57:40.446119 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.446078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6rh6\" (UniqueName: \"kubernetes.io/projected/c3ec9c16-6579-4bfe-be14-0405d5aa956a-kube-api-access-t6rh6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8\" (UID: \"c3ec9c16-6579-4bfe-be14-0405d5aa956a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8"
Jul 16 04:57:40.446286 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.446165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3ec9c16-6579-4bfe-be14-0405d5aa956a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8\" (UID: \"c3ec9c16-6579-4bfe-be14-0405d5aa956a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8"
Jul 16 04:57:40.446286 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.446226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3ec9c16-6579-4bfe-be14-0405d5aa956a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8\" (UID: \"c3ec9c16-6579-4bfe-be14-0405d5aa956a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8"
Jul 16 04:57:40.446626 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.446574    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3ec9c16-6579-4bfe-be14-0405d5aa956a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8\" (UID: \"c3ec9c16-6579-4bfe-be14-0405d5aa956a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8"
Jul 16 04:57:40.446671 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.446629    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3ec9c16-6579-4bfe-be14-0405d5aa956a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8\" (UID: \"c3ec9c16-6579-4bfe-be14-0405d5aa956a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8"
Jul 16 04:57:40.454505 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.454471    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6rh6\" (UniqueName: \"kubernetes.io/projected/c3ec9c16-6579-4bfe-be14-0405d5aa956a-kube-api-access-t6rh6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8\" (UID: \"c3ec9c16-6579-4bfe-be14-0405d5aa956a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8"
Jul 16 04:57:40.614673 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.614552    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8"
Jul 16 04:57:40.758472 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.758442    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8"]
Jul 16 04:57:40.760677 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:57:40.760643    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ec9c16_6579_4bfe_be14_0405d5aa956a.slice/crio-85c5606c4edce60d8193e82cbed7b9fed4d8d3caa715151bb3ed0119bca794d2 WatchSource:0}: Error finding container 85c5606c4edce60d8193e82cbed7b9fed4d8d3caa715151bb3ed0119bca794d2: Status 404 returned error can't find the container with id 85c5606c4edce60d8193e82cbed7b9fed4d8d3caa715151bb3ed0119bca794d2
Jul 16 04:57:40.944796 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:40.944700    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8" event={"ID":"c3ec9c16-6579-4bfe-be14-0405d5aa956a","Type":"ContainerStarted","Data":"85c5606c4edce60d8193e82cbed7b9fed4d8d3caa715151bb3ed0119bca794d2"}
Jul 16 04:57:46.977863 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:46.977756    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3ec9c16-6579-4bfe-be14-0405d5aa956a" containerID="f372cdae20a8490e4ec56b0b152c6b7883946ad123e23ec2b8d94613fb5b9674" exitCode=0
Jul 16 04:57:46.977863 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:46.977851    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8" event={"ID":"c3ec9c16-6579-4bfe-be14-0405d5aa956a","Type":"ContainerDied","Data":"f372cdae20a8490e4ec56b0b152c6b7883946ad123e23ec2b8d94613fb5b9674"}
Jul 16 04:57:49.994288 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:49.994253    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3ec9c16-6579-4bfe-be14-0405d5aa956a" containerID="2c0cb0ae76cc1a45f779acb7e14d86a5d2b1f1dece753a278a5babf629d46242" exitCode=0
Jul 16 04:57:49.994729 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:49.994333    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8" event={"ID":"c3ec9c16-6579-4bfe-be14-0405d5aa956a","Type":"ContainerDied","Data":"2c0cb0ae76cc1a45f779acb7e14d86a5d2b1f1dece753a278a5babf629d46242"}
Jul 16 04:57:58.037584 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:58.037541    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3ec9c16-6579-4bfe-be14-0405d5aa956a" containerID="af7ef42e013148974af5399f19a722a029ca1d6207538e62e97c8b269f431e50" exitCode=0
Jul 16 04:57:58.038021 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:58.037630    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8" event={"ID":"c3ec9c16-6579-4bfe-be14-0405d5aa956a","Type":"ContainerDied","Data":"af7ef42e013148974af5399f19a722a029ca1d6207538e62e97c8b269f431e50"}
Jul 16 04:57:59.178897 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:59.178866    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8"
Jul 16 04:57:59.326561 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:59.326482    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6rh6\" (UniqueName: \"kubernetes.io/projected/c3ec9c16-6579-4bfe-be14-0405d5aa956a-kube-api-access-t6rh6\") pod \"c3ec9c16-6579-4bfe-be14-0405d5aa956a\" (UID: \"c3ec9c16-6579-4bfe-be14-0405d5aa956a\") "
Jul 16 04:57:59.326848 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:59.326577    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3ec9c16-6579-4bfe-be14-0405d5aa956a-util\") pod \"c3ec9c16-6579-4bfe-be14-0405d5aa956a\" (UID: \"c3ec9c16-6579-4bfe-be14-0405d5aa956a\") "
Jul 16 04:57:59.326848 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:59.326612    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3ec9c16-6579-4bfe-be14-0405d5aa956a-bundle\") pod \"c3ec9c16-6579-4bfe-be14-0405d5aa956a\" (UID: \"c3ec9c16-6579-4bfe-be14-0405d5aa956a\") "
Jul 16 04:57:59.327275 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:59.327251    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ec9c16-6579-4bfe-be14-0405d5aa956a-bundle" (OuterVolumeSpecName: "bundle") pod "c3ec9c16-6579-4bfe-be14-0405d5aa956a" (UID: "c3ec9c16-6579-4bfe-be14-0405d5aa956a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 04:57:59.328861 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:59.328835    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ec9c16-6579-4bfe-be14-0405d5aa956a-kube-api-access-t6rh6" (OuterVolumeSpecName: "kube-api-access-t6rh6") pod "c3ec9c16-6579-4bfe-be14-0405d5aa956a" (UID: "c3ec9c16-6579-4bfe-be14-0405d5aa956a"). InnerVolumeSpecName "kube-api-access-t6rh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 04:57:59.331629 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:59.331581    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ec9c16-6579-4bfe-be14-0405d5aa956a-util" (OuterVolumeSpecName: "util") pod "c3ec9c16-6579-4bfe-be14-0405d5aa956a" (UID: "c3ec9c16-6579-4bfe-be14-0405d5aa956a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 04:57:59.427510 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:59.427467    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6rh6\" (UniqueName: \"kubernetes.io/projected/c3ec9c16-6579-4bfe-be14-0405d5aa956a-kube-api-access-t6rh6\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:57:59.427510 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:59.427503    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3ec9c16-6579-4bfe-be14-0405d5aa956a-util\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:57:59.427510 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:57:59.427513    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3ec9c16-6579-4bfe-be14-0405d5aa956a-bundle\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:58:00.049537 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:00.049508    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8"
Jul 16 04:58:00.049537 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:00.049523    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408mtg8" event={"ID":"c3ec9c16-6579-4bfe-be14-0405d5aa956a","Type":"ContainerDied","Data":"85c5606c4edce60d8193e82cbed7b9fed4d8d3caa715151bb3ed0119bca794d2"}
Jul 16 04:58:00.049788 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:00.049569    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85c5606c4edce60d8193e82cbed7b9fed4d8d3caa715151bb3ed0119bca794d2"
Jul 16 04:58:13.479521 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.479479    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q"]
Jul 16 04:58:13.479940 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.479828    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ec9c16-6579-4bfe-be14-0405d5aa956a" containerName="util"
Jul 16 04:58:13.479940 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.479839    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ec9c16-6579-4bfe-be14-0405d5aa956a" containerName="util"
Jul 16 04:58:13.479940 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.479850    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ec9c16-6579-4bfe-be14-0405d5aa956a" containerName="extract"
Jul 16 04:58:13.479940 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.479856    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ec9c16-6579-4bfe-be14-0405d5aa956a" containerName="extract"
Jul 16 04:58:13.479940 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.479870    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ec9c16-6579-4bfe-be14-0405d5aa956a" containerName="pull"
Jul 16 04:58:13.479940 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.479875    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ec9c16-6579-4bfe-be14-0405d5aa956a" containerName="pull"
Jul 16 04:58:13.479940 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.479935    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3ec9c16-6579-4bfe-be14-0405d5aa956a" containerName="extract"
Jul 16 04:58:13.484352 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.484329    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q"
Jul 16 04:58:13.486793 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.486766    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 04:58:13.486949 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.486815    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mz6kf\""
Jul 16 04:58:13.487915 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.487851    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 04:58:13.490800 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.490767    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q"]
Jul 16 04:58:13.548289 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.548249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43f94477-4724-4bb3-b299-3e7e2de27a03-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q\" (UID: \"43f94477-4724-4bb3-b299-3e7e2de27a03\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q"
Jul 16 04:58:13.548289 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.548296    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43f94477-4724-4bb3-b299-3e7e2de27a03-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q\" (UID: \"43f94477-4724-4bb3-b299-3e7e2de27a03\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q"
Jul 16 04:58:13.548531 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.548357    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rh42\" (UniqueName: \"kubernetes.io/projected/43f94477-4724-4bb3-b299-3e7e2de27a03-kube-api-access-7rh42\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q\" (UID: \"43f94477-4724-4bb3-b299-3e7e2de27a03\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q"
Jul 16 04:58:13.649356 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.649316    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43f94477-4724-4bb3-b299-3e7e2de27a03-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q\" (UID: \"43f94477-4724-4bb3-b299-3e7e2de27a03\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q"
Jul 16 04:58:13.649356 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.649368    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43f94477-4724-4bb3-b299-3e7e2de27a03-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q\" (UID: \"43f94477-4724-4bb3-b299-3e7e2de27a03\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q"
Jul 16 04:58:13.649672 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.649390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rh42\" (UniqueName: \"kubernetes.io/projected/43f94477-4724-4bb3-b299-3e7e2de27a03-kube-api-access-7rh42\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q\" (UID: \"43f94477-4724-4bb3-b299-3e7e2de27a03\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q"
Jul 16 04:58:13.649812 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.649786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43f94477-4724-4bb3-b299-3e7e2de27a03-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q\" (UID: \"43f94477-4724-4bb3-b299-3e7e2de27a03\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q"
Jul 16 04:58:13.649878 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.649829    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43f94477-4724-4bb3-b299-3e7e2de27a03-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q\" (UID: \"43f94477-4724-4bb3-b299-3e7e2de27a03\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q"
Jul 16 04:58:13.659896 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.659858    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rh42\" (UniqueName: \"kubernetes.io/projected/43f94477-4724-4bb3-b299-3e7e2de27a03-kube-api-access-7rh42\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q\" (UID: \"43f94477-4724-4bb3-b299-3e7e2de27a03\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q"
Jul 16 04:58:13.796254 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.796217    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q"
Jul 16 04:58:13.940750 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:13.940716    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q"]
Jul 16 04:58:13.944237 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:58:13.944191    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f94477_4724_4bb3_b299_3e7e2de27a03.slice/crio-7e0a108176bfce836a8c1c7f9f6cdaa3a14edfbd3dbb0234beae86120c7ca8a8 WatchSource:0}: Error finding container 7e0a108176bfce836a8c1c7f9f6cdaa3a14edfbd3dbb0234beae86120c7ca8a8: Status 404 returned error can't find the container with id 7e0a108176bfce836a8c1c7f9f6cdaa3a14edfbd3dbb0234beae86120c7ca8a8
Jul 16 04:58:14.113422 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:14.113381    2565 generic.go:356] "Generic (PLEG): container finished" podID="43f94477-4724-4bb3-b299-3e7e2de27a03" containerID="2ebfaa199f655e155d0f021da76437d7e831935ef980fc3f99ee505f4faf3617" exitCode=0
Jul 16 04:58:14.113633 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:14.113475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q" event={"ID":"43f94477-4724-4bb3-b299-3e7e2de27a03","Type":"ContainerDied","Data":"2ebfaa199f655e155d0f021da76437d7e831935ef980fc3f99ee505f4faf3617"}
Jul 16 04:58:14.113633 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:14.113533    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q" event={"ID":"43f94477-4724-4bb3-b299-3e7e2de27a03","Type":"ContainerStarted","Data":"7e0a108176bfce836a8c1c7f9f6cdaa3a14edfbd3dbb0234beae86120c7ca8a8"}
Jul 16 04:58:14.763780 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:14.763709    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zpwz2"]
Jul 16 04:58:14.766271 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:14.766244    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpwz2"
Jul 16 04:58:14.769036 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:14.768872    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 16 04:58:14.769036 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:14.768988    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nh6cr\""
Jul 16 04:58:14.769245 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:14.769068    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 16 04:58:14.774236 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:14.774174    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zpwz2"]
Jul 16 04:58:14.858000 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:14.857959    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv54c\" (UniqueName: \"kubernetes.io/projected/1ba0311a-a38d-48a0-86d2-ed3fba67a2cc-kube-api-access-tv54c\") pod \"cert-manager-cainjector-559cdd8c87-zpwz2\" (UID: \"1ba0311a-a38d-48a0-86d2-ed3fba67a2cc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpwz2"
Jul 16 04:58:14.858208 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:14.858110    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ba0311a-a38d-48a0-86d2-ed3fba67a2cc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zpwz2\" (UID: \"1ba0311a-a38d-48a0-86d2-ed3fba67a2cc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpwz2"
Jul 16 04:58:14.958790 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:14.958703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv54c\" (UniqueName: \"kubernetes.io/projected/1ba0311a-a38d-48a0-86d2-ed3fba67a2cc-kube-api-access-tv54c\") pod \"cert-manager-cainjector-559cdd8c87-zpwz2\" (UID: \"1ba0311a-a38d-48a0-86d2-ed3fba67a2cc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpwz2"
Jul 16 04:58:14.958790 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:14.958788    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ba0311a-a38d-48a0-86d2-ed3fba67a2cc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zpwz2\" (UID: \"1ba0311a-a38d-48a0-86d2-ed3fba67a2cc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpwz2"
Jul 16 04:58:14.966950 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:14.966919    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ba0311a-a38d-48a0-86d2-ed3fba67a2cc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zpwz2\" (UID: \"1ba0311a-a38d-48a0-86d2-ed3fba67a2cc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpwz2"
Jul 16 04:58:14.967215 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:14.967188    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv54c\" (UniqueName: \"kubernetes.io/projected/1ba0311a-a38d-48a0-86d2-ed3fba67a2cc-kube-api-access-tv54c\") pod \"cert-manager-cainjector-559cdd8c87-zpwz2\" (UID: \"1ba0311a-a38d-48a0-86d2-ed3fba67a2cc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpwz2"
Jul 16 04:58:15.086940 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:15.086901    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpwz2"
Jul 16 04:58:15.255441 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:15.255405    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zpwz2"]
Jul 16 04:58:15.258691 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:58:15.258653    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ba0311a_a38d_48a0_86d2_ed3fba67a2cc.slice/crio-9412266a8a107bf62b3993e1d3b163c63227195a7669e021a354fdc323bd7515 WatchSource:0}: Error finding container 9412266a8a107bf62b3993e1d3b163c63227195a7669e021a354fdc323bd7515: Status 404 returned error can't find the container with id 9412266a8a107bf62b3993e1d3b163c63227195a7669e021a354fdc323bd7515
Jul 16 04:58:16.126130 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:16.126095    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpwz2" event={"ID":"1ba0311a-a38d-48a0-86d2-ed3fba67a2cc","Type":"ContainerStarted","Data":"9412266a8a107bf62b3993e1d3b163c63227195a7669e021a354fdc323bd7515"}
Jul 16 04:58:16.128285 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:16.128240    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q" event={"ID":"43f94477-4724-4bb3-b299-3e7e2de27a03","Type":"ContainerStarted","Data":"a751a25ff3db72233d4ca382f262bdbc672910c3a9942c402699f6f350277af8"}
Jul 16 04:58:17.135908 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:17.135858    2565 generic.go:356] "Generic (PLEG): container finished" podID="43f94477-4724-4bb3-b299-3e7e2de27a03" containerID="a751a25ff3db72233d4ca382f262bdbc672910c3a9942c402699f6f350277af8" exitCode=0
Jul 16 04:58:17.142307 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:17.142275    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q" event={"ID":"43f94477-4724-4bb3-b299-3e7e2de27a03","Type":"ContainerDied","Data":"a751a25ff3db72233d4ca382f262bdbc672910c3a9942c402699f6f350277af8"}
Jul 16 04:58:18.143521 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:18.143475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q" event={"ID":"43f94477-4724-4bb3-b299-3e7e2de27a03","Type":"ContainerStarted","Data":"4bb89db1e51594ff67c79a4190b700aadbc2261c4897c2ed748cf221562784d2"}
Jul 16 04:58:18.161105 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:18.161037    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q" podStartSLOduration=3.238493437 podStartE2EDuration="5.161017878s" totalImagesPullingTime="1.922524441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:58:18.159548667 +0000 UTC m=+501.575503742" watchObservedRunningTime="2026-07-16 04:58:18.161017878 +0000 UTC m=+501.576972950"
Jul 16 04:58:19.150024 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:19.149981    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpwz2" event={"ID":"1ba0311a-a38d-48a0-86d2-ed3fba67a2cc","Type":"ContainerStarted","Data":"9dc606c4056e252f7ac4dd9917c207c1c7e0c1a74b7620c022833c3eb1ea8809"}
Jul 16 04:58:19.152208 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:19.152174    2565 generic.go:356] "Generic (PLEG): container finished" podID="43f94477-4724-4bb3-b299-3e7e2de27a03" containerID="4bb89db1e51594ff67c79a4190b700aadbc2261c4897c2ed748cf221562784d2" exitCode=0
Jul 16 04:58:19.152348 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:19.152246    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q" event={"ID":"43f94477-4724-4bb3-b299-3e7e2de27a03","Type":"ContainerDied","Data":"4bb89db1e51594ff67c79a4190b700aadbc2261c4897c2ed748cf221562784d2"}
Jul 16 04:58:19.165681 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:19.165622    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zpwz2" podStartSLOduration=2.14207603 podStartE2EDuration="5.165584225s" totalImagesPullingTime="3.023508195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:58:19.164500193 +0000 UTC m=+502.580455267" watchObservedRunningTime="2026-07-16 04:58:19.165584225 +0000 UTC m=+502.581539298"
Jul 16 04:58:20.295347 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:20.295305    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q"
Jul 16 04:58:20.413292 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:20.413190    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rh42\" (UniqueName: \"kubernetes.io/projected/43f94477-4724-4bb3-b299-3e7e2de27a03-kube-api-access-7rh42\") pod \"43f94477-4724-4bb3-b299-3e7e2de27a03\" (UID: \"43f94477-4724-4bb3-b299-3e7e2de27a03\") "
Jul 16 04:58:20.413292 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:20.413269    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43f94477-4724-4bb3-b299-3e7e2de27a03-bundle\") pod \"43f94477-4724-4bb3-b299-3e7e2de27a03\" (UID: \"43f94477-4724-4bb3-b299-3e7e2de27a03\") "
Jul 16 04:58:20.413483 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:20.413375    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43f94477-4724-4bb3-b299-3e7e2de27a03-util\") pod \"43f94477-4724-4bb3-b299-3e7e2de27a03\" (UID: \"43f94477-4724-4bb3-b299-3e7e2de27a03\") "
Jul 16 04:58:20.413756 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:20.413724    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f94477-4724-4bb3-b299-3e7e2de27a03-bundle" (OuterVolumeSpecName: "bundle") pod "43f94477-4724-4bb3-b299-3e7e2de27a03" (UID: "43f94477-4724-4bb3-b299-3e7e2de27a03"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 04:58:20.415902 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:20.415867    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f94477-4724-4bb3-b299-3e7e2de27a03-kube-api-access-7rh42" (OuterVolumeSpecName: "kube-api-access-7rh42") pod "43f94477-4724-4bb3-b299-3e7e2de27a03" (UID: "43f94477-4724-4bb3-b299-3e7e2de27a03"). InnerVolumeSpecName "kube-api-access-7rh42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 04:58:20.418382 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:20.418329    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f94477-4724-4bb3-b299-3e7e2de27a03-util" (OuterVolumeSpecName: "util") pod "43f94477-4724-4bb3-b299-3e7e2de27a03" (UID: "43f94477-4724-4bb3-b299-3e7e2de27a03"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 04:58:20.514213 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:20.514170    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43f94477-4724-4bb3-b299-3e7e2de27a03-util\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:58:20.514213 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:20.514206    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rh42\" (UniqueName: \"kubernetes.io/projected/43f94477-4724-4bb3-b299-3e7e2de27a03-kube-api-access-7rh42\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:58:20.514213 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:20.514217    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43f94477-4724-4bb3-b299-3e7e2de27a03-bundle\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:58:21.164080 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:21.164044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q" event={"ID":"43f94477-4724-4bb3-b299-3e7e2de27a03","Type":"ContainerDied","Data":"7e0a108176bfce836a8c1c7f9f6cdaa3a14edfbd3dbb0234beae86120c7ca8a8"}
Jul 16 04:58:21.164268 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:21.164112    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e0a108176bfce836a8c1c7f9f6cdaa3a14edfbd3dbb0234beae86120c7ca8a8"
Jul 16 04:58:21.164268 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:21.164062    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvs4q"
Jul 16 04:58:26.251509 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.251464    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ntkw"]
Jul 16 04:58:26.251937 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.251871    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f94477-4724-4bb3-b299-3e7e2de27a03" containerName="extract"
Jul 16 04:58:26.251937 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.251884    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f94477-4724-4bb3-b299-3e7e2de27a03" containerName="extract"
Jul 16 04:58:26.251937 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.251893    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f94477-4724-4bb3-b299-3e7e2de27a03" containerName="util"
Jul 16 04:58:26.251937 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.251899    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f94477-4724-4bb3-b299-3e7e2de27a03" containerName="util"
Jul 16 04:58:26.251937 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.251909    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f94477-4724-4bb3-b299-3e7e2de27a03" containerName="pull"
Jul 16 04:58:26.251937 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.251914    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f94477-4724-4bb3-b299-3e7e2de27a03" containerName="pull"
Jul 16 04:58:26.252145 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.251981    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f94477-4724-4bb3-b299-3e7e2de27a03" containerName="extract"
Jul 16 04:58:26.255870 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.255845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ntkw"
Jul 16 04:58:26.258540 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.258506    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 16 04:58:26.258700 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.258553    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 16 04:58:26.258700 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.258622    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-qq92n\""
Jul 16 04:58:26.264356 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.264328    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ntkw"]
Jul 16 04:58:26.363241 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.363190    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c3f6ce1-85a8-4bae-94fd-0052112bc5be-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4ntkw\" (UID: \"9c3f6ce1-85a8-4bae-94fd-0052112bc5be\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ntkw"
Jul 16 04:58:26.363433 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.363259    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfp2l\" (UniqueName: \"kubernetes.io/projected/9c3f6ce1-85a8-4bae-94fd-0052112bc5be-kube-api-access-dfp2l\") pod \"openshift-lws-operator-fd8ccff4c-4ntkw\" (UID: \"9c3f6ce1-85a8-4bae-94fd-0052112bc5be\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ntkw"
Jul 16 04:58:26.464620 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.464551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c3f6ce1-85a8-4bae-94fd-0052112bc5be-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4ntkw\" (UID: \"9c3f6ce1-85a8-4bae-94fd-0052112bc5be\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ntkw"
Jul 16 04:58:26.464777 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.464646    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfp2l\" (UniqueName: \"kubernetes.io/projected/9c3f6ce1-85a8-4bae-94fd-0052112bc5be-kube-api-access-dfp2l\") pod \"openshift-lws-operator-fd8ccff4c-4ntkw\" (UID: \"9c3f6ce1-85a8-4bae-94fd-0052112bc5be\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ntkw"
Jul 16 04:58:26.464953 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.464932    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c3f6ce1-85a8-4bae-94fd-0052112bc5be-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4ntkw\" (UID: \"9c3f6ce1-85a8-4bae-94fd-0052112bc5be\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ntkw"
Jul 16 04:58:26.478290 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.478252    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfp2l\" (UniqueName: \"kubernetes.io/projected/9c3f6ce1-85a8-4bae-94fd-0052112bc5be-kube-api-access-dfp2l\") pod \"openshift-lws-operator-fd8ccff4c-4ntkw\" (UID: \"9c3f6ce1-85a8-4bae-94fd-0052112bc5be\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ntkw"
Jul 16 04:58:26.566492 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.566382    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ntkw"
Jul 16 04:58:26.716398 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:26.716234    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ntkw"]
Jul 16 04:58:26.719471 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:58:26.719438    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c3f6ce1_85a8_4bae_94fd_0052112bc5be.slice/crio-cc44c2ac03697d4a5e03ccd5f090f247075a12810cf3994900313617e5ee426e WatchSource:0}: Error finding container cc44c2ac03697d4a5e03ccd5f090f247075a12810cf3994900313617e5ee426e: Status 404 returned error can't find the container with id cc44c2ac03697d4a5e03ccd5f090f247075a12810cf3994900313617e5ee426e
Jul 16 04:58:27.196383 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:27.196342    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ntkw" event={"ID":"9c3f6ce1-85a8-4bae-94fd-0052112bc5be","Type":"ContainerStarted","Data":"cc44c2ac03697d4a5e03ccd5f090f247075a12810cf3994900313617e5ee426e"}
Jul 16 04:58:29.209634 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:29.209562    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ntkw" event={"ID":"9c3f6ce1-85a8-4bae-94fd-0052112bc5be","Type":"ContainerStarted","Data":"cebd770832f346730339ac531d8d8f36b39428e7ab3b92a08b482740cae19a0a"}
Jul 16 04:58:29.225728 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:29.225674    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4ntkw" podStartSLOduration=1.453672278 podStartE2EDuration="3.225656682s" totalImagesPullingTime="1.771984404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:58:29.224093111 +0000 UTC m=+512.640048185" watchObservedRunningTime="2026-07-16 04:58:29.225656682 +0000 UTC m=+512.641611755"
Jul 16 04:58:31.855296 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:31.855250    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz"]
Jul 16 04:58:31.859201 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:31.859175    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz"
Jul 16 04:58:31.861759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:31.861730    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 04:58:31.861759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:31.861750    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 04:58:31.862665 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:31.862634    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mz6kf\""
Jul 16 04:58:31.866752 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:31.866720    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz"]
Jul 16 04:58:31.913879 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:31.913835    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz\" (UID: \"4ba7b5bd-f318-433f-81c9-cff30a87ff5e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz"
Jul 16 04:58:31.913879 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:31.913882    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz\" (UID: \"4ba7b5bd-f318-433f-81c9-cff30a87ff5e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz"
Jul 16 04:58:31.914150 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:31.913968    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lst56\" (UniqueName: \"kubernetes.io/projected/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-kube-api-access-lst56\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz\" (UID: \"4ba7b5bd-f318-433f-81c9-cff30a87ff5e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz"
Jul 16 04:58:32.015199 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.015151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz\" (UID: \"4ba7b5bd-f318-433f-81c9-cff30a87ff5e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz"
Jul 16 04:58:32.015199 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.015202    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz\" (UID: \"4ba7b5bd-f318-433f-81c9-cff30a87ff5e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz"
Jul 16 04:58:32.015492 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.015233    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lst56\" (UniqueName: \"kubernetes.io/projected/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-kube-api-access-lst56\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz\" (UID: \"4ba7b5bd-f318-433f-81c9-cff30a87ff5e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz"
Jul 16 04:58:32.015583 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.015559    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz\" (UID: \"4ba7b5bd-f318-433f-81c9-cff30a87ff5e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz"
Jul 16 04:58:32.015679 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.015626    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz\" (UID: \"4ba7b5bd-f318-433f-81c9-cff30a87ff5e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz"
Jul 16 04:58:32.023554 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.023526    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lst56\" (UniqueName: \"kubernetes.io/projected/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-kube-api-access-lst56\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz\" (UID: \"4ba7b5bd-f318-433f-81c9-cff30a87ff5e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz"
Jul 16 04:58:32.160730 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.160646    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vlkjv"]
Jul 16 04:58:32.164472 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.164443    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vlkjv"
Jul 16 04:58:32.166955 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.166928    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-nh4sz\""
Jul 16 04:58:32.170159 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.170094    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz"
Jul 16 04:58:32.173639 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.173152    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vlkjv"]
Jul 16 04:58:32.217731 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.217696    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk62q\" (UniqueName: \"kubernetes.io/projected/9612538f-4ec3-4b54-987c-5cde4012d492-kube-api-access-pk62q\") pod \"cert-manager-55b4d954d5-vlkjv\" (UID: \"9612538f-4ec3-4b54-987c-5cde4012d492\") " pod="cert-manager/cert-manager-55b4d954d5-vlkjv"
Jul 16 04:58:32.217911 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.217750    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9612538f-4ec3-4b54-987c-5cde4012d492-bound-sa-token\") pod \"cert-manager-55b4d954d5-vlkjv\" (UID: \"9612538f-4ec3-4b54-987c-5cde4012d492\") " pod="cert-manager/cert-manager-55b4d954d5-vlkjv"
Jul 16 04:58:32.318448 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.318410    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9612538f-4ec3-4b54-987c-5cde4012d492-bound-sa-token\") pod \"cert-manager-55b4d954d5-vlkjv\" (UID: \"9612538f-4ec3-4b54-987c-5cde4012d492\") " pod="cert-manager/cert-manager-55b4d954d5-vlkjv"
Jul 16 04:58:32.318676 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.318547    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk62q\" (UniqueName: \"kubernetes.io/projected/9612538f-4ec3-4b54-987c-5cde4012d492-kube-api-access-pk62q\") pod \"cert-manager-55b4d954d5-vlkjv\" (UID: \"9612538f-4ec3-4b54-987c-5cde4012d492\") " pod="cert-manager/cert-manager-55b4d954d5-vlkjv"
Jul 16 04:58:32.318979 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.318909    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz"]
Jul 16 04:58:32.321992 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:58:32.321962    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba7b5bd_f318_433f_81c9_cff30a87ff5e.slice/crio-5bfb86a356dfef922c07f35d9027f0a04cf8d0d6072afbe81dccfdbe9e0ad1bc WatchSource:0}: Error finding container 5bfb86a356dfef922c07f35d9027f0a04cf8d0d6072afbe81dccfdbe9e0ad1bc: Status 404 returned error can't find the container with id 5bfb86a356dfef922c07f35d9027f0a04cf8d0d6072afbe81dccfdbe9e0ad1bc
Jul 16 04:58:32.326466 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.326442    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9612538f-4ec3-4b54-987c-5cde4012d492-bound-sa-token\") pod \"cert-manager-55b4d954d5-vlkjv\" (UID: \"9612538f-4ec3-4b54-987c-5cde4012d492\") " pod="cert-manager/cert-manager-55b4d954d5-vlkjv"
Jul 16 04:58:32.326568 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.326480    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk62q\" (UniqueName: \"kubernetes.io/projected/9612538f-4ec3-4b54-987c-5cde4012d492-kube-api-access-pk62q\") pod \"cert-manager-55b4d954d5-vlkjv\" (UID: \"9612538f-4ec3-4b54-987c-5cde4012d492\") " pod="cert-manager/cert-manager-55b4d954d5-vlkjv"
Jul 16 04:58:32.476468 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.476429    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vlkjv"
Jul 16 04:58:32.618577 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:32.618540    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vlkjv"]
Jul 16 04:58:32.622344 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:58:32.622282    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9612538f_4ec3_4b54_987c_5cde4012d492.slice/crio-bb4d5eda49eb239fa09f43d4fc097b76bbbc804e2cc0e053fd8f827770ed6924 WatchSource:0}: Error finding container bb4d5eda49eb239fa09f43d4fc097b76bbbc804e2cc0e053fd8f827770ed6924: Status 404 returned error can't find the container with id bb4d5eda49eb239fa09f43d4fc097b76bbbc804e2cc0e053fd8f827770ed6924
Jul 16 04:58:33.230891 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:33.230845    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vlkjv" event={"ID":"9612538f-4ec3-4b54-987c-5cde4012d492","Type":"ContainerStarted","Data":"4102ee559a9fdafba37267c6471b97118a87e6410c643c5701d71046f16f4f6b"}
Jul 16 04:58:33.231390 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:33.230908    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vlkjv" event={"ID":"9612538f-4ec3-4b54-987c-5cde4012d492","Type":"ContainerStarted","Data":"bb4d5eda49eb239fa09f43d4fc097b76bbbc804e2cc0e053fd8f827770ed6924"}
Jul 16 04:58:33.232608 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:33.232566    2565 generic.go:356] "Generic (PLEG): container finished" podID="4ba7b5bd-f318-433f-81c9-cff30a87ff5e" containerID="670e139998e2bb54adeb1a16539c1e918b6fe757fbe2ee2d0bbca7f0c5a734dc" exitCode=0
Jul 16 04:58:33.232715 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:33.232631    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz" event={"ID":"4ba7b5bd-f318-433f-81c9-cff30a87ff5e","Type":"ContainerDied","Data":"670e139998e2bb54adeb1a16539c1e918b6fe757fbe2ee2d0bbca7f0c5a734dc"}
Jul 16 04:58:33.232715 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:33.232668    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz" event={"ID":"4ba7b5bd-f318-433f-81c9-cff30a87ff5e","Type":"ContainerStarted","Data":"5bfb86a356dfef922c07f35d9027f0a04cf8d0d6072afbe81dccfdbe9e0ad1bc"}
Jul 16 04:58:33.246765 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:33.246716    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-vlkjv" podStartSLOduration=1.246697434 podStartE2EDuration="1.246697434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:58:33.244827494 +0000 UTC m=+516.660782570" watchObservedRunningTime="2026-07-16 04:58:33.246697434 +0000 UTC m=+516.662652507"
Jul 16 04:58:34.239218 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:34.239182    2565 generic.go:356] "Generic (PLEG): container finished" podID="4ba7b5bd-f318-433f-81c9-cff30a87ff5e" containerID="ec6285d3615450d5ff1caf6c7d21be9cb84dbe897322620008d6c614d08d7f26" exitCode=0
Jul 16 04:58:34.239734 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:34.239264    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz" event={"ID":"4ba7b5bd-f318-433f-81c9-cff30a87ff5e","Type":"ContainerDied","Data":"ec6285d3615450d5ff1caf6c7d21be9cb84dbe897322620008d6c614d08d7f26"}
Jul 16 04:58:35.246462 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:35.246430    2565 generic.go:356] "Generic (PLEG): container finished" podID="4ba7b5bd-f318-433f-81c9-cff30a87ff5e" containerID="848a54284a0a9b84aeaddade74f249fc9582b4e1999243d7d8e9661a23867d51" exitCode=0
Jul 16 04:58:35.246864 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:35.246504    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz" event={"ID":"4ba7b5bd-f318-433f-81c9-cff30a87ff5e","Type":"ContainerDied","Data":"848a54284a0a9b84aeaddade74f249fc9582b4e1999243d7d8e9661a23867d51"}
Jul 16 04:58:36.392333 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:36.392300    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz"
Jul 16 04:58:36.457567 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:36.457527    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-bundle\") pod \"4ba7b5bd-f318-433f-81c9-cff30a87ff5e\" (UID: \"4ba7b5bd-f318-433f-81c9-cff30a87ff5e\") "
Jul 16 04:58:36.457773 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:36.457655    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-util\") pod \"4ba7b5bd-f318-433f-81c9-cff30a87ff5e\" (UID: \"4ba7b5bd-f318-433f-81c9-cff30a87ff5e\") "
Jul 16 04:58:36.457773 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:36.457676    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lst56\" (UniqueName: \"kubernetes.io/projected/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-kube-api-access-lst56\") pod \"4ba7b5bd-f318-433f-81c9-cff30a87ff5e\" (UID: \"4ba7b5bd-f318-433f-81c9-cff30a87ff5e\") "
Jul 16 04:58:36.458302 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:36.458276    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-bundle" (OuterVolumeSpecName: "bundle") pod "4ba7b5bd-f318-433f-81c9-cff30a87ff5e" (UID: "4ba7b5bd-f318-433f-81c9-cff30a87ff5e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 04:58:36.460024 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:36.459999    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-kube-api-access-lst56" (OuterVolumeSpecName: "kube-api-access-lst56") pod "4ba7b5bd-f318-433f-81c9-cff30a87ff5e" (UID: "4ba7b5bd-f318-433f-81c9-cff30a87ff5e"). InnerVolumeSpecName "kube-api-access-lst56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 04:58:36.465738 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:36.465693    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-util" (OuterVolumeSpecName: "util") pod "4ba7b5bd-f318-433f-81c9-cff30a87ff5e" (UID: "4ba7b5bd-f318-433f-81c9-cff30a87ff5e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 04:58:36.559343 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:36.559246    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-util\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:58:36.559343 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:36.559282    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lst56\" (UniqueName: \"kubernetes.io/projected/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-kube-api-access-lst56\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:58:36.559343 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:36.559293    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ba7b5bd-f318-433f-81c9-cff30a87ff5e-bundle\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 04:58:37.258525 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:37.258488    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz"
Jul 16 04:58:37.258525 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:37.258513    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpp8cz" event={"ID":"4ba7b5bd-f318-433f-81c9-cff30a87ff5e","Type":"ContainerDied","Data":"5bfb86a356dfef922c07f35d9027f0a04cf8d0d6072afbe81dccfdbe9e0ad1bc"}
Jul 16 04:58:37.258760 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:37.258569    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bfb86a356dfef922c07f35d9027f0a04cf8d0d6072afbe81dccfdbe9e0ad1bc"
Jul 16 04:58:49.128952 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.128863    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"]
Jul 16 04:58:49.130105 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.130072    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba7b5bd-f318-433f-81c9-cff30a87ff5e" containerName="pull"
Jul 16 04:58:49.130283 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.130271    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba7b5bd-f318-433f-81c9-cff30a87ff5e" containerName="pull"
Jul 16 04:58:49.130368 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.130359    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba7b5bd-f318-433f-81c9-cff30a87ff5e" containerName="extract"
Jul 16 04:58:49.130434 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.130426    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba7b5bd-f318-433f-81c9-cff30a87ff5e" containerName="extract"
Jul 16 04:58:49.130500 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.130491    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba7b5bd-f318-433f-81c9-cff30a87ff5e" containerName="util"
Jul 16 04:58:49.130557 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.130550    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba7b5bd-f318-433f-81c9-cff30a87ff5e" containerName="util"
Jul 16 04:58:49.130751 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.130741    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba7b5bd-f318-433f-81c9-cff30a87ff5e" containerName="extract"
Jul 16 04:58:49.134149 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.134116    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.136936 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.136896    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 16 04:58:49.137113 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.136986    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4rlvf\""
Jul 16 04:58:49.137113 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.137060    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 16 04:58:49.137208 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.137172    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 16 04:58:49.137270 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.137252    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 16 04:58:49.150792 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.150750    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"]
Jul 16 04:58:49.277836 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.277790    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df4937bc-c6ba-494d-83a0-b261dc5ef290-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dddb677c8-xhhkb\" (UID: \"df4937bc-c6ba-494d-83a0-b261dc5ef290\") " pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.277993 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.277864    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df4937bc-c6ba-494d-83a0-b261dc5ef290-tmp\") pod \"opendatahub-operator-controller-manager-6dddb677c8-xhhkb\" (UID: \"df4937bc-c6ba-494d-83a0-b261dc5ef290\") " pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.277993 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.277883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df4937bc-c6ba-494d-83a0-b261dc5ef290-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dddb677c8-xhhkb\" (UID: \"df4937bc-c6ba-494d-83a0-b261dc5ef290\") " pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.277993 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.277911    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df4937bc-c6ba-494d-83a0-b261dc5ef290-manifests\") pod \"opendatahub-operator-controller-manager-6dddb677c8-xhhkb\" (UID: \"df4937bc-c6ba-494d-83a0-b261dc5ef290\") " pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.277993 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.277971    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr6bg\" (UniqueName: \"kubernetes.io/projected/df4937bc-c6ba-494d-83a0-b261dc5ef290-kube-api-access-pr6bg\") pod \"opendatahub-operator-controller-manager-6dddb677c8-xhhkb\" (UID: \"df4937bc-c6ba-494d-83a0-b261dc5ef290\") " pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.379154 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.379055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df4937bc-c6ba-494d-83a0-b261dc5ef290-tmp\") pod \"opendatahub-operator-controller-manager-6dddb677c8-xhhkb\" (UID: \"df4937bc-c6ba-494d-83a0-b261dc5ef290\") " pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.379154 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.379138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df4937bc-c6ba-494d-83a0-b261dc5ef290-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dddb677c8-xhhkb\" (UID: \"df4937bc-c6ba-494d-83a0-b261dc5ef290\") " pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.379392 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.379182    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df4937bc-c6ba-494d-83a0-b261dc5ef290-manifests\") pod \"opendatahub-operator-controller-manager-6dddb677c8-xhhkb\" (UID: \"df4937bc-c6ba-494d-83a0-b261dc5ef290\") " pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.379392 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.379251    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr6bg\" (UniqueName: \"kubernetes.io/projected/df4937bc-c6ba-494d-83a0-b261dc5ef290-kube-api-access-pr6bg\") pod \"opendatahub-operator-controller-manager-6dddb677c8-xhhkb\" (UID: \"df4937bc-c6ba-494d-83a0-b261dc5ef290\") " pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.379392 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.379291    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df4937bc-c6ba-494d-83a0-b261dc5ef290-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dddb677c8-xhhkb\" (UID: \"df4937bc-c6ba-494d-83a0-b261dc5ef290\") " pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.379709 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.379683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df4937bc-c6ba-494d-83a0-b261dc5ef290-manifests\") pod \"opendatahub-operator-controller-manager-6dddb677c8-xhhkb\" (UID: \"df4937bc-c6ba-494d-83a0-b261dc5ef290\") " pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.382012 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.381981    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df4937bc-c6ba-494d-83a0-b261dc5ef290-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dddb677c8-xhhkb\" (UID: \"df4937bc-c6ba-494d-83a0-b261dc5ef290\") " pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.382217 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.382194    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df4937bc-c6ba-494d-83a0-b261dc5ef290-tmp\") pod \"opendatahub-operator-controller-manager-6dddb677c8-xhhkb\" (UID: \"df4937bc-c6ba-494d-83a0-b261dc5ef290\") " pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.382431 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.382413    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df4937bc-c6ba-494d-83a0-b261dc5ef290-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dddb677c8-xhhkb\" (UID: \"df4937bc-c6ba-494d-83a0-b261dc5ef290\") " pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.390239 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.390205    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr6bg\" (UniqueName: \"kubernetes.io/projected/df4937bc-c6ba-494d-83a0-b261dc5ef290-kube-api-access-pr6bg\") pod \"opendatahub-operator-controller-manager-6dddb677c8-xhhkb\" (UID: \"df4937bc-c6ba-494d-83a0-b261dc5ef290\") " pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.448393 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.448304    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:49.625339 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:49.625301    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"]
Jul 16 04:58:49.628534 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:58:49.628459    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4937bc_c6ba_494d_83a0_b261dc5ef290.slice/crio-740a0f46257d6265e7ebd0e1ea50492b8a9a3f5166daec4b76d210055ad4f94e WatchSource:0}: Error finding container 740a0f46257d6265e7ebd0e1ea50492b8a9a3f5166daec4b76d210055ad4f94e: Status 404 returned error can't find the container with id 740a0f46257d6265e7ebd0e1ea50492b8a9a3f5166daec4b76d210055ad4f94e
Jul 16 04:58:50.331184 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:50.331103    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb" event={"ID":"df4937bc-c6ba-494d-83a0-b261dc5ef290","Type":"ContainerStarted","Data":"740a0f46257d6265e7ebd0e1ea50492b8a9a3f5166daec4b76d210055ad4f94e"}
Jul 16 04:58:53.350878 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:53.350839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb" event={"ID":"df4937bc-c6ba-494d-83a0-b261dc5ef290","Type":"ContainerStarted","Data":"43c4f95d082452f5aa04b8aa3ff26edd1b45a8af1488504dac15e09e2067966b"}
Jul 16 04:58:54.357863 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:54.357825    2565 generic.go:356] "Generic (PLEG): container finished" podID="df4937bc-c6ba-494d-83a0-b261dc5ef290" containerID="43c4f95d082452f5aa04b8aa3ff26edd1b45a8af1488504dac15e09e2067966b" exitCode=0
Jul 16 04:58:54.357863 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:54.357869    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb" event={"ID":"df4937bc-c6ba-494d-83a0-b261dc5ef290","Type":"ContainerDied","Data":"43c4f95d082452f5aa04b8aa3ff26edd1b45a8af1488504dac15e09e2067966b"}
Jul 16 04:58:55.365863 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:55.365825    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb" event={"ID":"df4937bc-c6ba-494d-83a0-b261dc5ef290","Type":"ContainerStarted","Data":"4d911aa118e84d63cc86e94021156ffa81fc5255e728ca8ac7201599abaa7248"}
Jul 16 04:58:55.366349 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:55.365900    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:58:55.390759 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:58:55.390699    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb" podStartSLOduration=1.840540445 podStartE2EDuration="6.390677673s" totalImagesPullingTime="3.39607076s" totalInitContainerRuntime="1.154066468s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:58:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:58:55.387854616 +0000 UTC m=+538.803809702" watchObservedRunningTime="2026-07-16 04:58:55.390677673 +0000 UTC m=+538.806632824"
Jul 16 04:59:06.379823 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.379789    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6dddb677c8-xhhkb"
Jul 16 04:59:06.458608 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.458548    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"]
Jul 16 04:59:06.468072 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.468043    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"
Jul 16 04:59:06.470706 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.470658    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 16 04:59:06.470885 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.470728    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 16 04:59:06.470885 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.470676    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-chx7n\""
Jul 16 04:59:06.470885 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.470671    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 16 04:59:06.471780 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.471748    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"]
Jul 16 04:59:06.533983 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.533943    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/53181c67-a04a-4151-96eb-5ca9bcdcec30-metrics-cert\") pod \"lws-controller-manager-8647fcdd7d-b9cw4\" (UID: \"53181c67-a04a-4151-96eb-5ca9bcdcec30\") " pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"
Jul 16 04:59:06.533983 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.533988    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5p2x\" (UniqueName: \"kubernetes.io/projected/53181c67-a04a-4151-96eb-5ca9bcdcec30-kube-api-access-g5p2x\") pod \"lws-controller-manager-8647fcdd7d-b9cw4\" (UID: \"53181c67-a04a-4151-96eb-5ca9bcdcec30\") " pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"
Jul 16 04:59:06.534208 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.534107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/53181c67-a04a-4151-96eb-5ca9bcdcec30-manager-config\") pod \"lws-controller-manager-8647fcdd7d-b9cw4\" (UID: \"53181c67-a04a-4151-96eb-5ca9bcdcec30\") " pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"
Jul 16 04:59:06.534208 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.534173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53181c67-a04a-4151-96eb-5ca9bcdcec30-cert\") pod \"lws-controller-manager-8647fcdd7d-b9cw4\" (UID: \"53181c67-a04a-4151-96eb-5ca9bcdcec30\") " pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"
Jul 16 04:59:06.635008 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.634903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/53181c67-a04a-4151-96eb-5ca9bcdcec30-manager-config\") pod \"lws-controller-manager-8647fcdd7d-b9cw4\" (UID: \"53181c67-a04a-4151-96eb-5ca9bcdcec30\") " pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"
Jul 16 04:59:06.635008 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.634969    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53181c67-a04a-4151-96eb-5ca9bcdcec30-cert\") pod \"lws-controller-manager-8647fcdd7d-b9cw4\" (UID: \"53181c67-a04a-4151-96eb-5ca9bcdcec30\") " pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"
Jul 16 04:59:06.635236 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.635017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/53181c67-a04a-4151-96eb-5ca9bcdcec30-metrics-cert\") pod \"lws-controller-manager-8647fcdd7d-b9cw4\" (UID: \"53181c67-a04a-4151-96eb-5ca9bcdcec30\") " pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"
Jul 16 04:59:06.635236 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.635046    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5p2x\" (UniqueName: \"kubernetes.io/projected/53181c67-a04a-4151-96eb-5ca9bcdcec30-kube-api-access-g5p2x\") pod \"lws-controller-manager-8647fcdd7d-b9cw4\" (UID: \"53181c67-a04a-4151-96eb-5ca9bcdcec30\") " pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"
Jul 16 04:59:06.635606 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.635573    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/53181c67-a04a-4151-96eb-5ca9bcdcec30-manager-config\") pod \"lws-controller-manager-8647fcdd7d-b9cw4\" (UID: \"53181c67-a04a-4151-96eb-5ca9bcdcec30\") " pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"
Jul 16 04:59:06.637827 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.637796    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53181c67-a04a-4151-96eb-5ca9bcdcec30-cert\") pod \"lws-controller-manager-8647fcdd7d-b9cw4\" (UID: \"53181c67-a04a-4151-96eb-5ca9bcdcec30\") " pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"
Jul 16 04:59:06.637969 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.637830    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/53181c67-a04a-4151-96eb-5ca9bcdcec30-metrics-cert\") pod \"lws-controller-manager-8647fcdd7d-b9cw4\" (UID: \"53181c67-a04a-4151-96eb-5ca9bcdcec30\") " pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"
Jul 16 04:59:06.643403 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.643367    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5p2x\" (UniqueName: \"kubernetes.io/projected/53181c67-a04a-4151-96eb-5ca9bcdcec30-kube-api-access-g5p2x\") pod \"lws-controller-manager-8647fcdd7d-b9cw4\" (UID: \"53181c67-a04a-4151-96eb-5ca9bcdcec30\") " pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"
Jul 16 04:59:06.779802 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.779764    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"
Jul 16 04:59:06.930067 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:06.930037    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"]
Jul 16 04:59:06.933238 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:59:06.933200    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53181c67_a04a_4151_96eb_5ca9bcdcec30.slice/crio-0a82d150ce0ffe285086cf470ca5730668c8af09e700d337b6a4034f3f7d492e WatchSource:0}: Error finding container 0a82d150ce0ffe285086cf470ca5730668c8af09e700d337b6a4034f3f7d492e: Status 404 returned error can't find the container with id 0a82d150ce0ffe285086cf470ca5730668c8af09e700d337b6a4034f3f7d492e
Jul 16 04:59:07.437561 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:07.437525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4" event={"ID":"53181c67-a04a-4151-96eb-5ca9bcdcec30","Type":"ContainerStarted","Data":"0a82d150ce0ffe285086cf470ca5730668c8af09e700d337b6a4034f3f7d492e"}
Jul 16 04:59:08.729712 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.729622    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d"]
Jul 16 04:59:08.733431 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.733400    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d"
Jul 16 04:59:08.737544 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.737512    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 16 04:59:08.737544 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.737538    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8nd5d\""
Jul 16 04:59:08.737794 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.737517    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 04:59:08.737794 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.737512    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 16 04:59:08.737794 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.737514    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 04:59:08.744512 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.744482    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d"]
Jul 16 04:59:08.855642 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.855570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a972592d-555b-436e-bee8-2ae1a60f45e0-tmp\") pod \"kube-auth-proxy-7c7fc4f54f-mpm4d\" (UID: \"a972592d-555b-436e-bee8-2ae1a60f45e0\") " pod="openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d"
Jul 16 04:59:08.855859 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.855735    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a972592d-555b-436e-bee8-2ae1a60f45e0-tls-certs\") pod \"kube-auth-proxy-7c7fc4f54f-mpm4d\" (UID: \"a972592d-555b-436e-bee8-2ae1a60f45e0\") " pod="openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d"
Jul 16 04:59:08.855859 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.855779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlhqj\" (UniqueName: \"kubernetes.io/projected/a972592d-555b-436e-bee8-2ae1a60f45e0-kube-api-access-tlhqj\") pod \"kube-auth-proxy-7c7fc4f54f-mpm4d\" (UID: \"a972592d-555b-436e-bee8-2ae1a60f45e0\") " pod="openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d"
Jul 16 04:59:08.957357 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.957309    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a972592d-555b-436e-bee8-2ae1a60f45e0-tls-certs\") pod \"kube-auth-proxy-7c7fc4f54f-mpm4d\" (UID: \"a972592d-555b-436e-bee8-2ae1a60f45e0\") " pod="openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d"
Jul 16 04:59:08.957357 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.957354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlhqj\" (UniqueName: \"kubernetes.io/projected/a972592d-555b-436e-bee8-2ae1a60f45e0-kube-api-access-tlhqj\") pod \"kube-auth-proxy-7c7fc4f54f-mpm4d\" (UID: \"a972592d-555b-436e-bee8-2ae1a60f45e0\") " pod="openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d"
Jul 16 04:59:08.957671 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.957419    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a972592d-555b-436e-bee8-2ae1a60f45e0-tmp\") pod \"kube-auth-proxy-7c7fc4f54f-mpm4d\" (UID: \"a972592d-555b-436e-bee8-2ae1a60f45e0\") " pod="openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d"
Jul 16 04:59:08.960059 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.960023    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a972592d-555b-436e-bee8-2ae1a60f45e0-tmp\") pod \"kube-auth-proxy-7c7fc4f54f-mpm4d\" (UID: \"a972592d-555b-436e-bee8-2ae1a60f45e0\") " pod="openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d"
Jul 16 04:59:08.960202 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.960139    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a972592d-555b-436e-bee8-2ae1a60f45e0-tls-certs\") pod \"kube-auth-proxy-7c7fc4f54f-mpm4d\" (UID: \"a972592d-555b-436e-bee8-2ae1a60f45e0\") " pod="openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d"
Jul 16 04:59:08.971649 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:08.971617    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlhqj\" (UniqueName: \"kubernetes.io/projected/a972592d-555b-436e-bee8-2ae1a60f45e0-kube-api-access-tlhqj\") pod \"kube-auth-proxy-7c7fc4f54f-mpm4d\" (UID: \"a972592d-555b-436e-bee8-2ae1a60f45e0\") " pod="openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d"
Jul 16 04:59:09.045452 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:09.045414    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d"
Jul 16 04:59:09.226837 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:09.226796    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d"]
Jul 16 04:59:09.239902 ip-10-0-138-199 kubenswrapper[2565]: W0716 04:59:09.239869    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda972592d_555b_436e_bee8_2ae1a60f45e0.slice/crio-6def765bbb6adaa5e3fc001653192638596bdac1f238c7defc4012299f7353d7 WatchSource:0}: Error finding container 6def765bbb6adaa5e3fc001653192638596bdac1f238c7defc4012299f7353d7: Status 404 returned error can't find the container with id 6def765bbb6adaa5e3fc001653192638596bdac1f238c7defc4012299f7353d7
Jul 16 04:59:09.450673 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:09.450555    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d" event={"ID":"a972592d-555b-436e-bee8-2ae1a60f45e0","Type":"ContainerStarted","Data":"6def765bbb6adaa5e3fc001653192638596bdac1f238c7defc4012299f7353d7"}
Jul 16 04:59:09.452135 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:09.452099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4" event={"ID":"53181c67-a04a-4151-96eb-5ca9bcdcec30","Type":"ContainerStarted","Data":"6fef822284dc3b541683b5df330b2ad357804844e9fa9adb944267feb20962c6"}
Jul 16 04:59:09.452372 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:09.452350    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"
Jul 16 04:59:09.471390 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:09.471312    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4" podStartSLOduration=2.034279928 podStartE2EDuration="3.471286137s" totalImagesPullingTime="1.437006209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:59:09.468915201 +0000 UTC m=+552.884870273" watchObservedRunningTime="2026-07-16 04:59:09.471286137 +0000 UTC m=+552.887241211"
Jul 16 04:59:12.472650 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:12.472614    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d" event={"ID":"a972592d-555b-436e-bee8-2ae1a60f45e0","Type":"ContainerStarted","Data":"cd70a466d040a1d47e75202e80e6cc290ff78c2559e589b32b790c43e967f57e"}
Jul 16 04:59:12.490354 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:12.490286    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c7fc4f54f-mpm4d" podStartSLOduration=2.18295466 podStartE2EDuration="4.490262656s" totalImagesPullingTime="2.307307996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:59:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:59:12.487746387 +0000 UTC m=+555.903701462" watchObservedRunningTime="2026-07-16 04:59:12.490262656 +0000 UTC m=+555.906217731"
Jul 16 04:59:20.460528 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:20.460488    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8647fcdd7d-b9cw4"
Jul 16 04:59:57.330473 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:57.330336    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 04:59:57.335187 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:57.335171    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 04:59:57.366697 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:57.366670    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 04:59:57.371426 ip-10-0-138-199 kubenswrapper[2565]: I0716 04:59:57.371401    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:00:26.303206 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.303128    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv"]
Jul 16 05:00:26.306782 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.306762    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv"
Jul 16 05:00:26.309210 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.309177    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 05:00:26.309353 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.309191    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 05:00:26.310085 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.310062    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k4g7z\""
Jul 16 05:00:26.313545 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.313522    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv"]
Jul 16 05:00:26.380064 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.380032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82315cbd-b165-4552-8f50-301a9738f375-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv\" (UID: \"82315cbd-b165-4552-8f50-301a9738f375\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv"
Jul 16 05:00:26.380261 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.380108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44c8g\" (UniqueName: \"kubernetes.io/projected/82315cbd-b165-4552-8f50-301a9738f375-kube-api-access-44c8g\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv\" (UID: \"82315cbd-b165-4552-8f50-301a9738f375\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv"
Jul 16 05:00:26.380261 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.380131    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82315cbd-b165-4552-8f50-301a9738f375-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv\" (UID: \"82315cbd-b165-4552-8f50-301a9738f375\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv"
Jul 16 05:00:26.480757 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.480710    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44c8g\" (UniqueName: \"kubernetes.io/projected/82315cbd-b165-4552-8f50-301a9738f375-kube-api-access-44c8g\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv\" (UID: \"82315cbd-b165-4552-8f50-301a9738f375\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv"
Jul 16 05:00:26.480940 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.480766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82315cbd-b165-4552-8f50-301a9738f375-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv\" (UID: \"82315cbd-b165-4552-8f50-301a9738f375\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv"
Jul 16 05:00:26.480940 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.480824    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82315cbd-b165-4552-8f50-301a9738f375-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv\" (UID: \"82315cbd-b165-4552-8f50-301a9738f375\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv"
Jul 16 05:00:26.481178 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.481155    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82315cbd-b165-4552-8f50-301a9738f375-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv\" (UID: \"82315cbd-b165-4552-8f50-301a9738f375\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv"
Jul 16 05:00:26.481252 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.481224    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82315cbd-b165-4552-8f50-301a9738f375-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv\" (UID: \"82315cbd-b165-4552-8f50-301a9738f375\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv"
Jul 16 05:00:26.488989 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.488953    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44c8g\" (UniqueName: \"kubernetes.io/projected/82315cbd-b165-4552-8f50-301a9738f375-kube-api-access-44c8g\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv\" (UID: \"82315cbd-b165-4552-8f50-301a9738f375\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv"
Jul 16 05:00:26.618144 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.618054    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv"
Jul 16 05:00:26.783558 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.783531    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv"]
Jul 16 05:00:26.785861 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:00:26.785824    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82315cbd_b165_4552_8f50_301a9738f375.slice/crio-3a19159c17bf39d9bb40e91060587b98af9659d538d1a4b8932194f0b93e0ebd WatchSource:0}: Error finding container 3a19159c17bf39d9bb40e91060587b98af9659d538d1a4b8932194f0b93e0ebd: Status 404 returned error can't find the container with id 3a19159c17bf39d9bb40e91060587b98af9659d538d1a4b8932194f0b93e0ebd
Jul 16 05:00:26.855521 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.855483    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw"]
Jul 16 05:00:26.859790 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.859758    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv" event={"ID":"82315cbd-b165-4552-8f50-301a9738f375","Type":"ContainerStarted","Data":"3a19159c17bf39d9bb40e91060587b98af9659d538d1a4b8932194f0b93e0ebd"}
Jul 16 05:00:26.859933 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.859859    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw"
Jul 16 05:00:26.865102 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.865071    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw"]
Jul 16 05:00:26.883989 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.883948    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g97pv\" (UniqueName: \"kubernetes.io/projected/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-kube-api-access-g97pv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw\" (UID: \"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw"
Jul 16 05:00:26.883989 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.883992    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw\" (UID: \"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw"
Jul 16 05:00:26.884214 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.884135    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw\" (UID: \"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw"
Jul 16 05:00:26.984665 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.984585    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw\" (UID: \"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw"
Jul 16 05:00:26.984866 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.984721    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g97pv\" (UniqueName: \"kubernetes.io/projected/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-kube-api-access-g97pv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw\" (UID: \"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw"
Jul 16 05:00:26.984866 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.984755    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw\" (UID: \"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw"
Jul 16 05:00:26.985001 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.984975    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw\" (UID: \"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw"
Jul 16 05:00:26.985111 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.985090    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw\" (UID: \"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw"
Jul 16 05:00:26.993029 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:26.992994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g97pv\" (UniqueName: \"kubernetes.io/projected/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-kube-api-access-g97pv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw\" (UID: \"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw"
Jul 16 05:00:27.171974 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.171866    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw"
Jul 16 05:00:27.316451 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.316413    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw"]
Jul 16 05:00:27.320198 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:00:27.320166    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d4f5d2_7194_41fe_ae56_a0c65f5e5dc0.slice/crio-e47b4ed022cb1879d642305d303dcee6650cd172201e31b77748726443f9dec4 WatchSource:0}: Error finding container e47b4ed022cb1879d642305d303dcee6650cd172201e31b77748726443f9dec4: Status 404 returned error can't find the container with id e47b4ed022cb1879d642305d303dcee6650cd172201e31b77748726443f9dec4
Jul 16 05:00:27.450717 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.450679    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7"]
Jul 16 05:00:27.454672 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.454641    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7"
Jul 16 05:00:27.462172 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.462139    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7"]
Jul 16 05:00:27.488233 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.488190    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqjv4\" (UniqueName: \"kubernetes.io/projected/67b252d5-f05a-405e-8f7c-42c087adc22d-kube-api-access-fqjv4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7\" (UID: \"67b252d5-f05a-405e-8f7c-42c087adc22d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7"
Jul 16 05:00:27.488404 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.488269    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b252d5-f05a-405e-8f7c-42c087adc22d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7\" (UID: \"67b252d5-f05a-405e-8f7c-42c087adc22d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7"
Jul 16 05:00:27.488404 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.488295    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b252d5-f05a-405e-8f7c-42c087adc22d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7\" (UID: \"67b252d5-f05a-405e-8f7c-42c087adc22d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7"
Jul 16 05:00:27.589443 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.589407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b252d5-f05a-405e-8f7c-42c087adc22d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7\" (UID: \"67b252d5-f05a-405e-8f7c-42c087adc22d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7"
Jul 16 05:00:27.589443 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.589452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b252d5-f05a-405e-8f7c-42c087adc22d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7\" (UID: \"67b252d5-f05a-405e-8f7c-42c087adc22d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7"
Jul 16 05:00:27.589739 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.589526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqjv4\" (UniqueName: \"kubernetes.io/projected/67b252d5-f05a-405e-8f7c-42c087adc22d-kube-api-access-fqjv4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7\" (UID: \"67b252d5-f05a-405e-8f7c-42c087adc22d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7"
Jul 16 05:00:27.589956 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.589933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b252d5-f05a-405e-8f7c-42c087adc22d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7\" (UID: \"67b252d5-f05a-405e-8f7c-42c087adc22d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7"
Jul 16 05:00:27.589996 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.589968    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b252d5-f05a-405e-8f7c-42c087adc22d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7\" (UID: \"67b252d5-f05a-405e-8f7c-42c087adc22d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7"
Jul 16 05:00:27.597919 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.597890    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqjv4\" (UniqueName: \"kubernetes.io/projected/67b252d5-f05a-405e-8f7c-42c087adc22d-kube-api-access-fqjv4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7\" (UID: \"67b252d5-f05a-405e-8f7c-42c087adc22d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7"
Jul 16 05:00:27.809101 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.809064    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7"
Jul 16 05:00:27.857309 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.857271    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g"]
Jul 16 05:00:27.861479 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.861453    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g"
Jul 16 05:00:27.869836 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.869332    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g"]
Jul 16 05:00:27.873866 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.873140    2565 generic.go:356] "Generic (PLEG): container finished" podID="82315cbd-b165-4552-8f50-301a9738f375" containerID="12478db83d074e97ffdf3beeb58db4f5795b712331460cb577a44220432c9eca" exitCode=0
Jul 16 05:00:27.873866 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.873283    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv" event={"ID":"82315cbd-b165-4552-8f50-301a9738f375","Type":"ContainerDied","Data":"12478db83d074e97ffdf3beeb58db4f5795b712331460cb577a44220432c9eca"}
Jul 16 05:00:27.874144 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.874119    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 05:00:27.883964 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.883848    2565 generic.go:356] "Generic (PLEG): container finished" podID="b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0" containerID="61aa06b0318cc9d41c612e9cac8729a342018f2c5ebf0193a6c59380371b2de0" exitCode=0
Jul 16 05:00:27.883964 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.883923    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw" event={"ID":"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0","Type":"ContainerDied","Data":"61aa06b0318cc9d41c612e9cac8729a342018f2c5ebf0193a6c59380371b2de0"}
Jul 16 05:00:27.884195 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.883970    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw" event={"ID":"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0","Type":"ContainerStarted","Data":"e47b4ed022cb1879d642305d303dcee6650cd172201e31b77748726443f9dec4"}
Jul 16 05:00:27.892614 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.892559    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/560c797e-c2b3-41f5-917a-9dcfc341e9cb-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g\" (UID: \"560c797e-c2b3-41f5-917a-9dcfc341e9cb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g"
Jul 16 05:00:27.892794 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.892666    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvslt\" (UniqueName: \"kubernetes.io/projected/560c797e-c2b3-41f5-917a-9dcfc341e9cb-kube-api-access-gvslt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g\" (UID: \"560c797e-c2b3-41f5-917a-9dcfc341e9cb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g"
Jul 16 05:00:27.892794 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.892703    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/560c797e-c2b3-41f5-917a-9dcfc341e9cb-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g\" (UID: \"560c797e-c2b3-41f5-917a-9dcfc341e9cb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g"
Jul 16 05:00:27.966585 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.966553    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7"]
Jul 16 05:00:27.968571 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:00:27.968541    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b252d5_f05a_405e_8f7c_42c087adc22d.slice/crio-6230fa9b49022f7f4723d75e0ce118ed34be0dbd7c0112388dcb51ac4fb0512a WatchSource:0}: Error finding container 6230fa9b49022f7f4723d75e0ce118ed34be0dbd7c0112388dcb51ac4fb0512a: Status 404 returned error can't find the container with id 6230fa9b49022f7f4723d75e0ce118ed34be0dbd7c0112388dcb51ac4fb0512a
Jul 16 05:00:27.993982 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.993945    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/560c797e-c2b3-41f5-917a-9dcfc341e9cb-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g\" (UID: \"560c797e-c2b3-41f5-917a-9dcfc341e9cb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g"
Jul 16 05:00:27.994128 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.994030    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvslt\" (UniqueName: \"kubernetes.io/projected/560c797e-c2b3-41f5-917a-9dcfc341e9cb-kube-api-access-gvslt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g\" (UID: \"560c797e-c2b3-41f5-917a-9dcfc341e9cb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g"
Jul 16 05:00:27.994128 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.994078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/560c797e-c2b3-41f5-917a-9dcfc341e9cb-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g\" (UID: \"560c797e-c2b3-41f5-917a-9dcfc341e9cb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g"
Jul 16 05:00:27.994303 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.994285    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/560c797e-c2b3-41f5-917a-9dcfc341e9cb-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g\" (UID: \"560c797e-c2b3-41f5-917a-9dcfc341e9cb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g"
Jul 16 05:00:27.994360 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:27.994334    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/560c797e-c2b3-41f5-917a-9dcfc341e9cb-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g\" (UID: \"560c797e-c2b3-41f5-917a-9dcfc341e9cb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g"
Jul 16 05:00:28.003539 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:28.003512    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvslt\" (UniqueName: \"kubernetes.io/projected/560c797e-c2b3-41f5-917a-9dcfc341e9cb-kube-api-access-gvslt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g\" (UID: \"560c797e-c2b3-41f5-917a-9dcfc341e9cb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g"
Jul 16 05:00:28.186651 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:28.186568    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g"
Jul 16 05:00:28.331101 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:28.331075    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g"]
Jul 16 05:00:28.332971 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:00:28.332938    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560c797e_c2b3_41f5_917a_9dcfc341e9cb.slice/crio-f88daf71f520501ef4a0f496657e5b73d3ba4b1f15cc5c72060a9c49f87e3c32 WatchSource:0}: Error finding container f88daf71f520501ef4a0f496657e5b73d3ba4b1f15cc5c72060a9c49f87e3c32: Status 404 returned error can't find the container with id f88daf71f520501ef4a0f496657e5b73d3ba4b1f15cc5c72060a9c49f87e3c32
Jul 16 05:00:28.890451 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:28.890407    2565 generic.go:356] "Generic (PLEG): container finished" podID="67b252d5-f05a-405e-8f7c-42c087adc22d" containerID="90de06b6d89059fed8058430ab60f221f7e0d69e315d9c627ee4adcad84588e7" exitCode=0
Jul 16 05:00:28.890708 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:28.890503    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7" event={"ID":"67b252d5-f05a-405e-8f7c-42c087adc22d","Type":"ContainerDied","Data":"90de06b6d89059fed8058430ab60f221f7e0d69e315d9c627ee4adcad84588e7"}
Jul 16 05:00:28.890708 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:28.890620    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7" event={"ID":"67b252d5-f05a-405e-8f7c-42c087adc22d","Type":"ContainerStarted","Data":"6230fa9b49022f7f4723d75e0ce118ed34be0dbd7c0112388dcb51ac4fb0512a"}
Jul 16 05:00:28.892115 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:28.892095    2565 generic.go:356] "Generic (PLEG): container finished" podID="560c797e-c2b3-41f5-917a-9dcfc341e9cb" containerID="ea62cf2aeb6e1cf7c73d358f82826e1f68f5be453bf2d875aecb4c4f4cfd2c14" exitCode=0
Jul 16 05:00:28.892206 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:28.892186    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g" event={"ID":"560c797e-c2b3-41f5-917a-9dcfc341e9cb","Type":"ContainerDied","Data":"ea62cf2aeb6e1cf7c73d358f82826e1f68f5be453bf2d875aecb4c4f4cfd2c14"}
Jul 16 05:00:28.892248 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:28.892232    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g" event={"ID":"560c797e-c2b3-41f5-917a-9dcfc341e9cb","Type":"ContainerStarted","Data":"f88daf71f520501ef4a0f496657e5b73d3ba4b1f15cc5c72060a9c49f87e3c32"}
Jul 16 05:00:29.901366 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:29.901330    2565 generic.go:356] "Generic (PLEG): container finished" podID="b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0" containerID="3bf2117f0c2a49d37ee9f488533a2230ad6d62cc5f572badd517b897031d711c" exitCode=0
Jul 16 05:00:29.901761 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:29.901407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw" event={"ID":"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0","Type":"ContainerDied","Data":"3bf2117f0c2a49d37ee9f488533a2230ad6d62cc5f572badd517b897031d711c"}
Jul 16 05:00:30.909342 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:30.909301    2565 generic.go:356] "Generic (PLEG): container finished" podID="82315cbd-b165-4552-8f50-301a9738f375" containerID="b7d761db32cb3bdacb7fff86717eb67dfeef53b0dd3c0464c7f10bdeb5123efd" exitCode=0
Jul 16 05:00:30.909874 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:30.909379    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv" event={"ID":"82315cbd-b165-4552-8f50-301a9738f375","Type":"ContainerDied","Data":"b7d761db32cb3bdacb7fff86717eb67dfeef53b0dd3c0464c7f10bdeb5123efd"}
Jul 16 05:00:30.911956 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:30.911933    2565 generic.go:356] "Generic (PLEG): container finished" podID="b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0" containerID="4686c12fed7cab13fd72b9fdf63ae6f07c2a59d5a72265481886df557aa9ec6c" exitCode=0
Jul 16 05:00:30.912079 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:30.912012    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw" event={"ID":"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0","Type":"ContainerDied","Data":"4686c12fed7cab13fd72b9fdf63ae6f07c2a59d5a72265481886df557aa9ec6c"}
Jul 16 05:00:30.914082 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:30.914057    2565 generic.go:356] "Generic (PLEG): container finished" podID="67b252d5-f05a-405e-8f7c-42c087adc22d" containerID="1479c772a80612eeebbe9edc7e0be79c2b8663b0209c449829cbbf29df13188c" exitCode=0
Jul 16 05:00:30.914238 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:30.914101    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7" event={"ID":"67b252d5-f05a-405e-8f7c-42c087adc22d","Type":"ContainerDied","Data":"1479c772a80612eeebbe9edc7e0be79c2b8663b0209c449829cbbf29df13188c"}
Jul 16 05:00:31.921344 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:31.921307    2565 generic.go:356] "Generic (PLEG): container finished" podID="560c797e-c2b3-41f5-917a-9dcfc341e9cb" containerID="34ab2f39931ed383165150fa9c12f1bcad8de28622ff83d12fa7be480cf70a38" exitCode=0
Jul 16 05:00:31.921865 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:31.921368    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g" event={"ID":"560c797e-c2b3-41f5-917a-9dcfc341e9cb","Type":"ContainerDied","Data":"34ab2f39931ed383165150fa9c12f1bcad8de28622ff83d12fa7be480cf70a38"}
Jul 16 05:00:31.923755 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:31.923729    2565 generic.go:356] "Generic (PLEG): container finished" podID="82315cbd-b165-4552-8f50-301a9738f375" containerID="9ae2172a572980c556d7487a8719a7998ba52a9d76f3db37279b9cfd96b9eca1" exitCode=0
Jul 16 05:00:31.923868 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:31.923793    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv" event={"ID":"82315cbd-b165-4552-8f50-301a9738f375","Type":"ContainerDied","Data":"9ae2172a572980c556d7487a8719a7998ba52a9d76f3db37279b9cfd96b9eca1"}
Jul 16 05:00:31.926090 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:31.926067    2565 generic.go:356] "Generic (PLEG): container finished" podID="67b252d5-f05a-405e-8f7c-42c087adc22d" containerID="3ac1e81b0efb595d2f26a747e12ed8d70918fc83615a4350d79d3925d311ed30" exitCode=0
Jul 16 05:00:31.926209 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:31.926151    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7" event={"ID":"67b252d5-f05a-405e-8f7c-42c087adc22d","Type":"ContainerDied","Data":"3ac1e81b0efb595d2f26a747e12ed8d70918fc83615a4350d79d3925d311ed30"}
Jul 16 05:00:32.072342 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:32.072311    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw"
Jul 16 05:00:32.134889 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:32.134853    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-util\") pod \"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0\" (UID: \"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0\") "
Jul 16 05:00:32.135044 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:32.134921    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-bundle\") pod \"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0\" (UID: \"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0\") "
Jul 16 05:00:32.135044 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:32.134956    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g97pv\" (UniqueName: \"kubernetes.io/projected/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-kube-api-access-g97pv\") pod \"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0\" (UID: \"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0\") "
Jul 16 05:00:32.135437 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:32.135413    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-bundle" (OuterVolumeSpecName: "bundle") pod "b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0" (UID: "b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:00:32.137475 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:32.137437    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-kube-api-access-g97pv" (OuterVolumeSpecName: "kube-api-access-g97pv") pod "b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0" (UID: "b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0"). InnerVolumeSpecName "kube-api-access-g97pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:00:32.140662 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:32.140619    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-util" (OuterVolumeSpecName: "util") pod "b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0" (UID: "b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:00:32.236405 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:32.236372    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-util\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:00:32.236405 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:32.236407    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-bundle\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:00:32.236698 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:32.236420    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g97pv\" (UniqueName: \"kubernetes.io/projected/b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0-kube-api-access-g97pv\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:00:32.934290 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:32.934255    2565 generic.go:356] "Generic (PLEG): container finished" podID="560c797e-c2b3-41f5-917a-9dcfc341e9cb" containerID="fc3a02bd94290cc5934b4843d2098fd4f11dd14895a0b96791913c7cb5c3b35a" exitCode=0
Jul 16 05:00:32.934711 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:32.934347    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g" event={"ID":"560c797e-c2b3-41f5-917a-9dcfc341e9cb","Type":"ContainerDied","Data":"fc3a02bd94290cc5934b4843d2098fd4f11dd14895a0b96791913c7cb5c3b35a"}
Jul 16 05:00:32.936438 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:32.936411    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw"
Jul 16 05:00:32.936438 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:32.936421    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw" event={"ID":"b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0","Type":"ContainerDied","Data":"e47b4ed022cb1879d642305d303dcee6650cd172201e31b77748726443f9dec4"}
Jul 16 05:00:32.936629 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:32.936465    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e47b4ed022cb1879d642305d303dcee6650cd172201e31b77748726443f9dec4"
Jul 16 05:00:33.103768 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.103736    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv"
Jul 16 05:00:33.133239 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.133209    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7"
Jul 16 05:00:33.149322 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.149293    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44c8g\" (UniqueName: \"kubernetes.io/projected/82315cbd-b165-4552-8f50-301a9738f375-kube-api-access-44c8g\") pod \"82315cbd-b165-4552-8f50-301a9738f375\" (UID: \"82315cbd-b165-4552-8f50-301a9738f375\") "
Jul 16 05:00:33.149505 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.149358    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82315cbd-b165-4552-8f50-301a9738f375-util\") pod \"82315cbd-b165-4552-8f50-301a9738f375\" (UID: \"82315cbd-b165-4552-8f50-301a9738f375\") "
Jul 16 05:00:33.149505 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.149420    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82315cbd-b165-4552-8f50-301a9738f375-bundle\") pod \"82315cbd-b165-4552-8f50-301a9738f375\" (UID: \"82315cbd-b165-4552-8f50-301a9738f375\") "
Jul 16 05:00:33.150007 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.149970    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82315cbd-b165-4552-8f50-301a9738f375-bundle" (OuterVolumeSpecName: "bundle") pod "82315cbd-b165-4552-8f50-301a9738f375" (UID: "82315cbd-b165-4552-8f50-301a9738f375"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:00:33.152557 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.152519    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82315cbd-b165-4552-8f50-301a9738f375-kube-api-access-44c8g" (OuterVolumeSpecName: "kube-api-access-44c8g") pod "82315cbd-b165-4552-8f50-301a9738f375" (UID: "82315cbd-b165-4552-8f50-301a9738f375"). InnerVolumeSpecName "kube-api-access-44c8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:00:33.156774 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.156736    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82315cbd-b165-4552-8f50-301a9738f375-util" (OuterVolumeSpecName: "util") pod "82315cbd-b165-4552-8f50-301a9738f375" (UID: "82315cbd-b165-4552-8f50-301a9738f375"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:00:33.250811 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.250773    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b252d5-f05a-405e-8f7c-42c087adc22d-util\") pod \"67b252d5-f05a-405e-8f7c-42c087adc22d\" (UID: \"67b252d5-f05a-405e-8f7c-42c087adc22d\") "
Jul 16 05:00:33.250998 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.250840    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b252d5-f05a-405e-8f7c-42c087adc22d-bundle\") pod \"67b252d5-f05a-405e-8f7c-42c087adc22d\" (UID: \"67b252d5-f05a-405e-8f7c-42c087adc22d\") "
Jul 16 05:00:33.250998 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.250910    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqjv4\" (UniqueName: \"kubernetes.io/projected/67b252d5-f05a-405e-8f7c-42c087adc22d-kube-api-access-fqjv4\") pod \"67b252d5-f05a-405e-8f7c-42c087adc22d\" (UID: \"67b252d5-f05a-405e-8f7c-42c087adc22d\") "
Jul 16 05:00:33.251154 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.251127    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44c8g\" (UniqueName: \"kubernetes.io/projected/82315cbd-b165-4552-8f50-301a9738f375-kube-api-access-44c8g\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:00:33.251154 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.251146    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82315cbd-b165-4552-8f50-301a9738f375-util\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:00:33.251272 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.251158    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82315cbd-b165-4552-8f50-301a9738f375-bundle\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:00:33.251544 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.251519    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b252d5-f05a-405e-8f7c-42c087adc22d-bundle" (OuterVolumeSpecName: "bundle") pod "67b252d5-f05a-405e-8f7c-42c087adc22d" (UID: "67b252d5-f05a-405e-8f7c-42c087adc22d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:00:33.253244 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.253220    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b252d5-f05a-405e-8f7c-42c087adc22d-kube-api-access-fqjv4" (OuterVolumeSpecName: "kube-api-access-fqjv4") pod "67b252d5-f05a-405e-8f7c-42c087adc22d" (UID: "67b252d5-f05a-405e-8f7c-42c087adc22d"). InnerVolumeSpecName "kube-api-access-fqjv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:00:33.256352 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.256324    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b252d5-f05a-405e-8f7c-42c087adc22d-util" (OuterVolumeSpecName: "util") pod "67b252d5-f05a-405e-8f7c-42c087adc22d" (UID: "67b252d5-f05a-405e-8f7c-42c087adc22d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:00:33.352560 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.352515    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b252d5-f05a-405e-8f7c-42c087adc22d-bundle\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:00:33.352560 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.352550    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqjv4\" (UniqueName: \"kubernetes.io/projected/67b252d5-f05a-405e-8f7c-42c087adc22d-kube-api-access-fqjv4\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:00:33.352560 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.352560    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b252d5-f05a-405e-8f7c-42c087adc22d-util\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:00:33.944280 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.944244    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7"
Jul 16 05:00:33.944280 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.944267    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7" event={"ID":"67b252d5-f05a-405e-8f7c-42c087adc22d","Type":"ContainerDied","Data":"6230fa9b49022f7f4723d75e0ce118ed34be0dbd7c0112388dcb51ac4fb0512a"}
Jul 16 05:00:33.944768 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.944326    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6230fa9b49022f7f4723d75e0ce118ed34be0dbd7c0112388dcb51ac4fb0512a"
Jul 16 05:00:33.946316 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.946290    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv"
Jul 16 05:00:33.946442 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.946311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv" event={"ID":"82315cbd-b165-4552-8f50-301a9738f375","Type":"ContainerDied","Data":"3a19159c17bf39d9bb40e91060587b98af9659d538d1a4b8932194f0b93e0ebd"}
Jul 16 05:00:33.946442 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:33.946362    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a19159c17bf39d9bb40e91060587b98af9659d538d1a4b8932194f0b93e0ebd"
Jul 16 05:00:34.076088 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:34.076059    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g"
Jul 16 05:00:34.158401 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:34.158365    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/560c797e-c2b3-41f5-917a-9dcfc341e9cb-util\") pod \"560c797e-c2b3-41f5-917a-9dcfc341e9cb\" (UID: \"560c797e-c2b3-41f5-917a-9dcfc341e9cb\") "
Jul 16 05:00:34.158620 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:34.158422    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvslt\" (UniqueName: \"kubernetes.io/projected/560c797e-c2b3-41f5-917a-9dcfc341e9cb-kube-api-access-gvslt\") pod \"560c797e-c2b3-41f5-917a-9dcfc341e9cb\" (UID: \"560c797e-c2b3-41f5-917a-9dcfc341e9cb\") "
Jul 16 05:00:34.158620 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:34.158445    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/560c797e-c2b3-41f5-917a-9dcfc341e9cb-bundle\") pod \"560c797e-c2b3-41f5-917a-9dcfc341e9cb\" (UID: \"560c797e-c2b3-41f5-917a-9dcfc341e9cb\") "
Jul 16 05:00:34.159076 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:34.159035    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560c797e-c2b3-41f5-917a-9dcfc341e9cb-bundle" (OuterVolumeSpecName: "bundle") pod "560c797e-c2b3-41f5-917a-9dcfc341e9cb" (UID: "560c797e-c2b3-41f5-917a-9dcfc341e9cb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:00:34.160791 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:34.160758    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560c797e-c2b3-41f5-917a-9dcfc341e9cb-kube-api-access-gvslt" (OuterVolumeSpecName: "kube-api-access-gvslt") pod "560c797e-c2b3-41f5-917a-9dcfc341e9cb" (UID: "560c797e-c2b3-41f5-917a-9dcfc341e9cb"). InnerVolumeSpecName "kube-api-access-gvslt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:00:34.163123 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:34.163098    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560c797e-c2b3-41f5-917a-9dcfc341e9cb-util" (OuterVolumeSpecName: "util") pod "560c797e-c2b3-41f5-917a-9dcfc341e9cb" (UID: "560c797e-c2b3-41f5-917a-9dcfc341e9cb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:00:34.260663 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:34.260633    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvslt\" (UniqueName: \"kubernetes.io/projected/560c797e-c2b3-41f5-917a-9dcfc341e9cb-kube-api-access-gvslt\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:00:34.260663 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:34.260662    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/560c797e-c2b3-41f5-917a-9dcfc341e9cb-bundle\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:00:34.260862 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:34.260675    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/560c797e-c2b3-41f5-917a-9dcfc341e9cb-util\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:00:34.955101 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:34.955070    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g"
Jul 16 05:00:34.955521 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:34.955097    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g" event={"ID":"560c797e-c2b3-41f5-917a-9dcfc341e9cb","Type":"ContainerDied","Data":"f88daf71f520501ef4a0f496657e5b73d3ba4b1f15cc5c72060a9c49f87e3c32"}
Jul 16 05:00:34.955521 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:34.955157    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f88daf71f520501ef4a0f496657e5b73d3ba4b1f15cc5c72060a9c49f87e3c32"
Jul 16 05:00:48.164123 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:00:48.164086    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-655d596b95-w4j6j"]
Jul 16 05:01:13.186934 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.186888    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-655d596b95-w4j6j" podUID="cad72dda-df2f-4541-91f3-573a5b98dbf7" containerName="console" containerID="cri-o://105fa0cc8387be3b51cf45dbd8f1fcd223ff085f2fed40b57391f610ad9c9332" gracePeriod=15
Jul 16 05:01:13.449282 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.449209    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 05:01:13.517108 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.517071    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-oauth-serving-cert\") pod \"cad72dda-df2f-4541-91f3-573a5b98dbf7\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") "
Jul 16 05:01:13.517108 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.517111    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wwq8\" (UniqueName: \"kubernetes.io/projected/cad72dda-df2f-4541-91f3-573a5b98dbf7-kube-api-access-6wwq8\") pod \"cad72dda-df2f-4541-91f3-573a5b98dbf7\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") "
Jul 16 05:01:13.517108 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.517128    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-serving-cert\") pod \"cad72dda-df2f-4541-91f3-573a5b98dbf7\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") "
Jul 16 05:01:13.517420 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.517157    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-config\") pod \"cad72dda-df2f-4541-91f3-573a5b98dbf7\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") "
Jul 16 05:01:13.517420 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.517198    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-trusted-ca-bundle\") pod \"cad72dda-df2f-4541-91f3-573a5b98dbf7\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") "
Jul 16 05:01:13.517420 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.517233    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-oauth-config\") pod \"cad72dda-df2f-4541-91f3-573a5b98dbf7\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") "
Jul 16 05:01:13.517420 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.517291    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-service-ca\") pod \"cad72dda-df2f-4541-91f3-573a5b98dbf7\" (UID: \"cad72dda-df2f-4541-91f3-573a5b98dbf7\") "
Jul 16 05:01:13.517764 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.517515    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cad72dda-df2f-4541-91f3-573a5b98dbf7" (UID: "cad72dda-df2f-4541-91f3-573a5b98dbf7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 05:01:13.517764 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.517747    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-config" (OuterVolumeSpecName: "console-config") pod "cad72dda-df2f-4541-91f3-573a5b98dbf7" (UID: "cad72dda-df2f-4541-91f3-573a5b98dbf7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 05:01:13.517946 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.517809    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cad72dda-df2f-4541-91f3-573a5b98dbf7" (UID: "cad72dda-df2f-4541-91f3-573a5b98dbf7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 05:01:13.517946 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.517838    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-service-ca" (OuterVolumeSpecName: "service-ca") pod "cad72dda-df2f-4541-91f3-573a5b98dbf7" (UID: "cad72dda-df2f-4541-91f3-573a5b98dbf7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 05:01:13.519501 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.519481    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cad72dda-df2f-4541-91f3-573a5b98dbf7" (UID: "cad72dda-df2f-4541-91f3-573a5b98dbf7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 05:01:13.520041 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.520018    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cad72dda-df2f-4541-91f3-573a5b98dbf7" (UID: "cad72dda-df2f-4541-91f3-573a5b98dbf7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 05:01:13.520130 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.520036    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad72dda-df2f-4541-91f3-573a5b98dbf7-kube-api-access-6wwq8" (OuterVolumeSpecName: "kube-api-access-6wwq8") pod "cad72dda-df2f-4541-91f3-573a5b98dbf7" (UID: "cad72dda-df2f-4541-91f3-573a5b98dbf7"). InnerVolumeSpecName "kube-api-access-6wwq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:01:13.618770 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.618712    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-oauth-serving-cert\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:01:13.618770 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.618761    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wwq8\" (UniqueName: \"kubernetes.io/projected/cad72dda-df2f-4541-91f3-573a5b98dbf7-kube-api-access-6wwq8\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:01:13.618770 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.618774    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-serving-cert\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:01:13.618770 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.618782    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-config\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:01:13.618770 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.618791    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-trusted-ca-bundle\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:01:13.619084 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.618799    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cad72dda-df2f-4541-91f3-573a5b98dbf7-console-oauth-config\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:01:13.619084 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:13.618808    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cad72dda-df2f-4541-91f3-573a5b98dbf7-service-ca\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:01:14.174904 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:14.174869    2565 generic.go:356] "Generic (PLEG): container finished" podID="cad72dda-df2f-4541-91f3-573a5b98dbf7" containerID="105fa0cc8387be3b51cf45dbd8f1fcd223ff085f2fed40b57391f610ad9c9332" exitCode=0
Jul 16 05:01:14.175090 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:14.174914    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655d596b95-w4j6j"
Jul 16 05:01:14.175090 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:14.174960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655d596b95-w4j6j" event={"ID":"cad72dda-df2f-4541-91f3-573a5b98dbf7","Type":"ContainerDied","Data":"105fa0cc8387be3b51cf45dbd8f1fcd223ff085f2fed40b57391f610ad9c9332"}
Jul 16 05:01:14.175090 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:14.175011    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655d596b95-w4j6j" event={"ID":"cad72dda-df2f-4541-91f3-573a5b98dbf7","Type":"ContainerDied","Data":"f92d3862e4f22d8425cb8251e710ed21a96ec7347ee5bcf03fa4d2d6e27dfac7"}
Jul 16 05:01:14.175090 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:14.175031    2565 scope.go:117] "RemoveContainer" containerID="105fa0cc8387be3b51cf45dbd8f1fcd223ff085f2fed40b57391f610ad9c9332"
Jul 16 05:01:14.185429 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:14.185404    2565 scope.go:117] "RemoveContainer" containerID="105fa0cc8387be3b51cf45dbd8f1fcd223ff085f2fed40b57391f610ad9c9332"
Jul 16 05:01:14.185790 ip-10-0-138-199 kubenswrapper[2565]: E0716 05:01:14.185767    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"105fa0cc8387be3b51cf45dbd8f1fcd223ff085f2fed40b57391f610ad9c9332\": container with ID starting with 105fa0cc8387be3b51cf45dbd8f1fcd223ff085f2fed40b57391f610ad9c9332 not found: ID does not exist" containerID="105fa0cc8387be3b51cf45dbd8f1fcd223ff085f2fed40b57391f610ad9c9332"
Jul 16 05:01:14.185861 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:14.185799    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105fa0cc8387be3b51cf45dbd8f1fcd223ff085f2fed40b57391f610ad9c9332"} err="failed to get container status \"105fa0cc8387be3b51cf45dbd8f1fcd223ff085f2fed40b57391f610ad9c9332\": rpc error: code = NotFound desc = could not find container \"105fa0cc8387be3b51cf45dbd8f1fcd223ff085f2fed40b57391f610ad9c9332\": container with ID starting with 105fa0cc8387be3b51cf45dbd8f1fcd223ff085f2fed40b57391f610ad9c9332 not found: ID does not exist"
Jul 16 05:01:14.198853 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:14.198820    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-655d596b95-w4j6j"]
Jul 16 05:01:14.200952 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:14.200926    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-655d596b95-w4j6j"]
Jul 16 05:01:15.144049 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:15.143999    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cad72dda-df2f-4541-91f3-573a5b98dbf7" path="/var/lib/kubelet/pods/cad72dda-df2f-4541-91f3-573a5b98dbf7/volumes"
Jul 16 05:01:56.125192 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125097    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125453    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b252d5-f05a-405e-8f7c-42c087adc22d" containerName="util"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125464    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b252d5-f05a-405e-8f7c-42c087adc22d" containerName="util"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125475    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0" containerName="util"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125481    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0" containerName="util"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125492    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82315cbd-b165-4552-8f50-301a9738f375" containerName="util"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125500    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="82315cbd-b165-4552-8f50-301a9738f375" containerName="util"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125506    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0" containerName="pull"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125511    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0" containerName="pull"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125516    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b252d5-f05a-405e-8f7c-42c087adc22d" containerName="pull"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125521    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b252d5-f05a-405e-8f7c-42c087adc22d" containerName="pull"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125526    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82315cbd-b165-4552-8f50-301a9738f375" containerName="extract"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125531    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="82315cbd-b165-4552-8f50-301a9738f375" containerName="extract"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125536    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b252d5-f05a-405e-8f7c-42c087adc22d" containerName="extract"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125540    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b252d5-f05a-405e-8f7c-42c087adc22d" containerName="extract"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125548    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82315cbd-b165-4552-8f50-301a9738f375" containerName="pull"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125553    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="82315cbd-b165-4552-8f50-301a9738f375" containerName="pull"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125559    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0" containerName="extract"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125564    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0" containerName="extract"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125575    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cad72dda-df2f-4541-91f3-573a5b98dbf7" containerName="console"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125580    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cad72dda-df2f-4541-91f3-573a5b98dbf7" containerName="console"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125609    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560c797e-c2b3-41f5-917a-9dcfc341e9cb" containerName="util"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125614    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="560c797e-c2b3-41f5-917a-9dcfc341e9cb" containerName="util"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125619    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560c797e-c2b3-41f5-917a-9dcfc341e9cb" containerName="pull"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125624    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="560c797e-c2b3-41f5-917a-9dcfc341e9cb" containerName="pull"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125630    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560c797e-c2b3-41f5-917a-9dcfc341e9cb" containerName="extract"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125635    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="560c797e-c2b3-41f5-917a-9dcfc341e9cb" containerName="extract"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125693    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cad72dda-df2f-4541-91f3-573a5b98dbf7" containerName="console"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125700    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="67b252d5-f05a-405e-8f7c-42c087adc22d" containerName="extract"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125708    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="82315cbd-b165-4552-8f50-301a9738f375" containerName="extract"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125714    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0" containerName="extract"
Jul 16 05:01:56.125743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.125723    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="560c797e-c2b3-41f5-917a-9dcfc341e9cb" containerName="extract"
Jul 16 05:01:56.128737 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.128717    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-78gs7"
Jul 16 05:01:56.131295 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.131260    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 05:01:56.131475 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.131302    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 05:01:56.132366 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.132342    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k4g7z\""
Jul 16 05:01:56.132506 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.132373    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 16 05:01:56.139544 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.139513    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:01:56.165433 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.165387    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:01:56.210574 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.210530    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwpg8\" (UniqueName: \"kubernetes.io/projected/65fca02e-3e0b-4e0d-a5a3-60ca94b6f1cb-kube-api-access-kwpg8\") pod \"limitador-limitador-54d6d988d5-78gs7\" (UID: \"65fca02e-3e0b-4e0d-a5a3-60ca94b6f1cb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-78gs7"
Jul 16 05:01:56.210781 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.210631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/65fca02e-3e0b-4e0d-a5a3-60ca94b6f1cb-config-file\") pod \"limitador-limitador-54d6d988d5-78gs7\" (UID: \"65fca02e-3e0b-4e0d-a5a3-60ca94b6f1cb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-78gs7"
Jul 16 05:01:56.311086 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.311055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwpg8\" (UniqueName: \"kubernetes.io/projected/65fca02e-3e0b-4e0d-a5a3-60ca94b6f1cb-kube-api-access-kwpg8\") pod \"limitador-limitador-54d6d988d5-78gs7\" (UID: \"65fca02e-3e0b-4e0d-a5a3-60ca94b6f1cb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-78gs7"
Jul 16 05:01:56.311086 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.311090    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/65fca02e-3e0b-4e0d-a5a3-60ca94b6f1cb-config-file\") pod \"limitador-limitador-54d6d988d5-78gs7\" (UID: \"65fca02e-3e0b-4e0d-a5a3-60ca94b6f1cb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-78gs7"
Jul 16 05:01:56.311747 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.311728    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/65fca02e-3e0b-4e0d-a5a3-60ca94b6f1cb-config-file\") pod \"limitador-limitador-54d6d988d5-78gs7\" (UID: \"65fca02e-3e0b-4e0d-a5a3-60ca94b6f1cb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-78gs7"
Jul 16 05:01:56.321998 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.321970    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwpg8\" (UniqueName: \"kubernetes.io/projected/65fca02e-3e0b-4e0d-a5a3-60ca94b6f1cb-kube-api-access-kwpg8\") pod \"limitador-limitador-54d6d988d5-78gs7\" (UID: \"65fca02e-3e0b-4e0d-a5a3-60ca94b6f1cb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-78gs7"
Jul 16 05:01:56.441028 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.440930    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-78gs7"
Jul 16 05:01:56.498796 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.498755    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-g9wxs"]
Jul 16 05:01:56.502739 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.502705    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g9wxs"
Jul 16 05:01:56.505474 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.505417    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jwp4b\""
Jul 16 05:01:56.507467 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.507439    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-g9wxs"]
Jul 16 05:01:56.597385 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.597353    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:01:56.598907 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:01:56.598879    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65fca02e_3e0b_4e0d_a5a3_60ca94b6f1cb.slice/crio-1a15be69b2f0e465dd33caa7f56fdcd82ce1568c959c3659869933682d32a76d WatchSource:0}: Error finding container 1a15be69b2f0e465dd33caa7f56fdcd82ce1568c959c3659869933682d32a76d: Status 404 returned error can't find the container with id 1a15be69b2f0e465dd33caa7f56fdcd82ce1568c959c3659869933682d32a76d
Jul 16 05:01:56.614279 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.614239    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vtnr\" (UniqueName: \"kubernetes.io/projected/63db68df-06ec-4e27-9775-260f5d8873cd-kube-api-access-8vtnr\") pod \"authorino-58cd955675-g9wxs\" (UID: \"63db68df-06ec-4e27-9775-260f5d8873cd\") " pod="kuadrant-system/authorino-58cd955675-g9wxs"
Jul 16 05:01:56.614484 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.614468    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-l9vwr"]
Jul 16 05:01:56.617989 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.617965    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-l9vwr"
Jul 16 05:01:56.624459 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.624400    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-l9vwr"]
Jul 16 05:01:56.715435 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.715334    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vtnr\" (UniqueName: \"kubernetes.io/projected/63db68df-06ec-4e27-9775-260f5d8873cd-kube-api-access-8vtnr\") pod \"authorino-58cd955675-g9wxs\" (UID: \"63db68df-06ec-4e27-9775-260f5d8873cd\") " pod="kuadrant-system/authorino-58cd955675-g9wxs"
Jul 16 05:01:56.715435 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.715394    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j6lt\" (UniqueName: \"kubernetes.io/projected/d01f4c10-70eb-4641-af56-80dd3dc04587-kube-api-access-9j6lt\") pod \"authorino-75d556bd77-l9vwr\" (UID: \"d01f4c10-70eb-4641-af56-80dd3dc04587\") " pod="kuadrant-system/authorino-75d556bd77-l9vwr"
Jul 16 05:01:56.726637 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.726606    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vtnr\" (UniqueName: \"kubernetes.io/projected/63db68df-06ec-4e27-9775-260f5d8873cd-kube-api-access-8vtnr\") pod \"authorino-58cd955675-g9wxs\" (UID: \"63db68df-06ec-4e27-9775-260f5d8873cd\") " pod="kuadrant-system/authorino-58cd955675-g9wxs"
Jul 16 05:01:56.816804 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.816758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6lt\" (UniqueName: \"kubernetes.io/projected/d01f4c10-70eb-4641-af56-80dd3dc04587-kube-api-access-9j6lt\") pod \"authorino-75d556bd77-l9vwr\" (UID: \"d01f4c10-70eb-4641-af56-80dd3dc04587\") " pod="kuadrant-system/authorino-75d556bd77-l9vwr"
Jul 16 05:01:56.818561 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.818539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g9wxs"
Jul 16 05:01:56.824397 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.824373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j6lt\" (UniqueName: \"kubernetes.io/projected/d01f4c10-70eb-4641-af56-80dd3dc04587-kube-api-access-9j6lt\") pod \"authorino-75d556bd77-l9vwr\" (UID: \"d01f4c10-70eb-4641-af56-80dd3dc04587\") " pod="kuadrant-system/authorino-75d556bd77-l9vwr"
Jul 16 05:01:56.931069 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.931033    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-l9vwr"
Jul 16 05:01:56.964561 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:56.964526    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-g9wxs"]
Jul 16 05:01:56.966231 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:01:56.966146    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63db68df_06ec_4e27_9775_260f5d8873cd.slice/crio-80ae1986778a4b71c5ee7cdf3f208f40014cd3bd5b8d4399abd729cd43ea505f WatchSource:0}: Error finding container 80ae1986778a4b71c5ee7cdf3f208f40014cd3bd5b8d4399abd729cd43ea505f: Status 404 returned error can't find the container with id 80ae1986778a4b71c5ee7cdf3f208f40014cd3bd5b8d4399abd729cd43ea505f
Jul 16 05:01:57.079739 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:57.079647    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-l9vwr"]
Jul 16 05:01:57.085848 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:01:57.085817    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd01f4c10_70eb_4641_af56_80dd3dc04587.slice/crio-70406672f56e30126e010312ece5c9c0b77151b345ba6aed9a7e0aced016478b WatchSource:0}: Error finding container 70406672f56e30126e010312ece5c9c0b77151b345ba6aed9a7e0aced016478b: Status 404 returned error can't find the container with id 70406672f56e30126e010312ece5c9c0b77151b345ba6aed9a7e0aced016478b
Jul 16 05:01:57.418854 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:57.418745    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-l9vwr" event={"ID":"d01f4c10-70eb-4641-af56-80dd3dc04587","Type":"ContainerStarted","Data":"70406672f56e30126e010312ece5c9c0b77151b345ba6aed9a7e0aced016478b"}
Jul 16 05:01:57.423417 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:57.423341    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g9wxs" event={"ID":"63db68df-06ec-4e27-9775-260f5d8873cd","Type":"ContainerStarted","Data":"80ae1986778a4b71c5ee7cdf3f208f40014cd3bd5b8d4399abd729cd43ea505f"}
Jul 16 05:01:57.426101 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:01:57.425987    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-78gs7" event={"ID":"65fca02e-3e0b-4e0d-a5a3-60ca94b6f1cb","Type":"ContainerStarted","Data":"1a15be69b2f0e465dd33caa7f56fdcd82ce1568c959c3659869933682d32a76d"}
Jul 16 05:02:02.465788 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:02.465747    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-78gs7" event={"ID":"65fca02e-3e0b-4e0d-a5a3-60ca94b6f1cb","Type":"ContainerStarted","Data":"ebf6a032338eccd7ad06c4ca93848257d927143b0e5035de9273a509326d59b3"}
Jul 16 05:02:02.466268 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:02.465953    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-78gs7"
Jul 16 05:02:02.467466 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:02.467444    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-l9vwr" event={"ID":"d01f4c10-70eb-4641-af56-80dd3dc04587","Type":"ContainerStarted","Data":"e0529a8142a50c14ebc8886de3d4ee391d0911284747c55fad706f03bbd5f4d6"}
Jul 16 05:02:02.469042 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:02.469016    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g9wxs" event={"ID":"63db68df-06ec-4e27-9775-260f5d8873cd","Type":"ContainerStarted","Data":"bc2623417cf90299431f22f6aa13de71ae24fdca4522386ef1c7215ed3420297"}
Jul 16 05:02:02.483093 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:02.483044    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-78gs7" podStartSLOduration=1.313456677 podStartE2EDuration="6.483026925s" totalImagesPullingTime="5.169570248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:02:02.480370704 +0000 UTC m=+725.896325779" watchObservedRunningTime="2026-07-16 05:02:02.483026925 +0000 UTC m=+725.898981997"
Jul 16 05:02:02.495007 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:02.494951    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-g9wxs" podStartSLOduration=1.738863227 podStartE2EDuration="6.494934518s" totalImagesPullingTime="4.756071291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:02:02.493197551 +0000 UTC m=+725.909152625" watchObservedRunningTime="2026-07-16 05:02:02.494934518 +0000 UTC m=+725.910889592"
Jul 16 05:02:02.507176 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:02.507105    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-l9vwr" podStartSLOduration=1.836227698 podStartE2EDuration="6.507078253s" totalImagesPullingTime="4.670850555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:02:02.506415856 +0000 UTC m=+725.922370935" watchObservedRunningTime="2026-07-16 05:02:02.507078253 +0000 UTC m=+725.923033326"
Jul 16 05:02:02.532351 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:02.532315    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-g9wxs"]
Jul 16 05:02:04.481182 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:04.481129    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-g9wxs" podUID="63db68df-06ec-4e27-9775-260f5d8873cd" containerName="authorino" containerID="cri-o://bc2623417cf90299431f22f6aa13de71ae24fdca4522386ef1c7215ed3420297" gracePeriod=30
Jul 16 05:02:04.734354 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:04.734323    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g9wxs"
Jul 16 05:02:04.798295 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:04.798252    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vtnr\" (UniqueName: \"kubernetes.io/projected/63db68df-06ec-4e27-9775-260f5d8873cd-kube-api-access-8vtnr\") pod \"63db68df-06ec-4e27-9775-260f5d8873cd\" (UID: \"63db68df-06ec-4e27-9775-260f5d8873cd\") "
Jul 16 05:02:04.800606 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:04.800561    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63db68df-06ec-4e27-9775-260f5d8873cd-kube-api-access-8vtnr" (OuterVolumeSpecName: "kube-api-access-8vtnr") pod "63db68df-06ec-4e27-9775-260f5d8873cd" (UID: "63db68df-06ec-4e27-9775-260f5d8873cd"). InnerVolumeSpecName "kube-api-access-8vtnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:02:04.898977 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:04.898910    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vtnr\" (UniqueName: \"kubernetes.io/projected/63db68df-06ec-4e27-9775-260f5d8873cd-kube-api-access-8vtnr\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:02:05.488439 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:05.488396    2565 generic.go:356] "Generic (PLEG): container finished" podID="63db68df-06ec-4e27-9775-260f5d8873cd" containerID="bc2623417cf90299431f22f6aa13de71ae24fdca4522386ef1c7215ed3420297" exitCode=0
Jul 16 05:02:05.488439 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:05.488442    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g9wxs" event={"ID":"63db68df-06ec-4e27-9775-260f5d8873cd","Type":"ContainerDied","Data":"bc2623417cf90299431f22f6aa13de71ae24fdca4522386ef1c7215ed3420297"}
Jul 16 05:02:05.488990 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:05.488478    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g9wxs" event={"ID":"63db68df-06ec-4e27-9775-260f5d8873cd","Type":"ContainerDied","Data":"80ae1986778a4b71c5ee7cdf3f208f40014cd3bd5b8d4399abd729cd43ea505f"}
Jul 16 05:02:05.488990 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:05.488500    2565 scope.go:117] "RemoveContainer" containerID="bc2623417cf90299431f22f6aa13de71ae24fdca4522386ef1c7215ed3420297"
Jul 16 05:02:05.488990 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:05.488446    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g9wxs"
Jul 16 05:02:05.498166 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:05.498142    2565 scope.go:117] "RemoveContainer" containerID="bc2623417cf90299431f22f6aa13de71ae24fdca4522386ef1c7215ed3420297"
Jul 16 05:02:05.498505 ip-10-0-138-199 kubenswrapper[2565]: E0716 05:02:05.498483    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2623417cf90299431f22f6aa13de71ae24fdca4522386ef1c7215ed3420297\": container with ID starting with bc2623417cf90299431f22f6aa13de71ae24fdca4522386ef1c7215ed3420297 not found: ID does not exist" containerID="bc2623417cf90299431f22f6aa13de71ae24fdca4522386ef1c7215ed3420297"
Jul 16 05:02:05.498547 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:05.498515    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2623417cf90299431f22f6aa13de71ae24fdca4522386ef1c7215ed3420297"} err="failed to get container status \"bc2623417cf90299431f22f6aa13de71ae24fdca4522386ef1c7215ed3420297\": rpc error: code = NotFound desc = could not find container \"bc2623417cf90299431f22f6aa13de71ae24fdca4522386ef1c7215ed3420297\": container with ID starting with bc2623417cf90299431f22f6aa13de71ae24fdca4522386ef1c7215ed3420297 not found: ID does not exist"
Jul 16 05:02:05.504251 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:05.504219    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-g9wxs"]
Jul 16 05:02:05.506556 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:05.506531    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-g9wxs"]
Jul 16 05:02:07.143239 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:07.143198    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63db68df-06ec-4e27-9775-260f5d8873cd" path="/var/lib/kubelet/pods/63db68df-06ec-4e27-9775-260f5d8873cd/volumes"
Jul 16 05:02:13.477140 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:13.477105    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-78gs7"
Jul 16 05:02:30.254170 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:30.254124    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nr25p"]
Jul 16 05:02:30.254761 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:30.254736    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63db68df-06ec-4e27-9775-260f5d8873cd" containerName="authorino"
Jul 16 05:02:30.254854 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:30.254765    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="63db68df-06ec-4e27-9775-260f5d8873cd" containerName="authorino"
Jul 16 05:02:30.254906 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:30.254878    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="63db68df-06ec-4e27-9775-260f5d8873cd" containerName="authorino"
Jul 16 05:02:30.261390 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:30.261356    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-nr25p"
Jul 16 05:02:30.264377 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:30.264345    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nr25p"]
Jul 16 05:02:30.333668 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:30.333623    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdpdw\" (UniqueName: \"kubernetes.io/projected/d3ae8ef1-57a9-4eef-bd6b-4122335629a2-kube-api-access-vdpdw\") pod \"authorino-5d4b67c446-nr25p\" (UID: \"d3ae8ef1-57a9-4eef-bd6b-4122335629a2\") " pod="kuadrant-system/authorino-5d4b67c446-nr25p"
Jul 16 05:02:30.434927 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:30.434889    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdpdw\" (UniqueName: \"kubernetes.io/projected/d3ae8ef1-57a9-4eef-bd6b-4122335629a2-kube-api-access-vdpdw\") pod \"authorino-5d4b67c446-nr25p\" (UID: \"d3ae8ef1-57a9-4eef-bd6b-4122335629a2\") " pod="kuadrant-system/authorino-5d4b67c446-nr25p"
Jul 16 05:02:30.442652 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:30.442586    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdpdw\" (UniqueName: \"kubernetes.io/projected/d3ae8ef1-57a9-4eef-bd6b-4122335629a2-kube-api-access-vdpdw\") pod \"authorino-5d4b67c446-nr25p\" (UID: \"d3ae8ef1-57a9-4eef-bd6b-4122335629a2\") " pod="kuadrant-system/authorino-5d4b67c446-nr25p"
Jul 16 05:02:30.493415 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:30.493381    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nr25p"]
Jul 16 05:02:30.493731 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:30.493717    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-nr25p"
Jul 16 05:02:30.639778 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:30.639753    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nr25p"]
Jul 16 05:02:30.641879 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:02:30.641850    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3ae8ef1_57a9_4eef_bd6b_4122335629a2.slice/crio-48cc5952dc788edd546d1790cd631ade480859b3e3d719932c0902a798243371 WatchSource:0}: Error finding container 48cc5952dc788edd546d1790cd631ade480859b3e3d719932c0902a798243371: Status 404 returned error can't find the container with id 48cc5952dc788edd546d1790cd631ade480859b3e3d719932c0902a798243371
Jul 16 05:02:31.631654 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:31.631581    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-nr25p" event={"ID":"d3ae8ef1-57a9-4eef-bd6b-4122335629a2","Type":"ContainerStarted","Data":"0c706fa0440fa713d9862ed4bc8b87c59bad7ec01f43c0f96fb50cc4c2b8de14"}
Jul 16 05:02:31.632081 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:31.631666    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-nr25p" event={"ID":"d3ae8ef1-57a9-4eef-bd6b-4122335629a2","Type":"ContainerStarted","Data":"48cc5952dc788edd546d1790cd631ade480859b3e3d719932c0902a798243371"}
Jul 16 05:02:31.632081 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:31.631709    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-nr25p" podUID="d3ae8ef1-57a9-4eef-bd6b-4122335629a2" containerName="authorino" containerID="cri-o://0c706fa0440fa713d9862ed4bc8b87c59bad7ec01f43c0f96fb50cc4c2b8de14" gracePeriod=30
Jul 16 05:02:31.894424 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:31.894352    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-nr25p"
Jul 16 05:02:31.946997 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:31.946962    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdpdw\" (UniqueName: \"kubernetes.io/projected/d3ae8ef1-57a9-4eef-bd6b-4122335629a2-kube-api-access-vdpdw\") pod \"d3ae8ef1-57a9-4eef-bd6b-4122335629a2\" (UID: \"d3ae8ef1-57a9-4eef-bd6b-4122335629a2\") "
Jul 16 05:02:31.949271 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:31.949238    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ae8ef1-57a9-4eef-bd6b-4122335629a2-kube-api-access-vdpdw" (OuterVolumeSpecName: "kube-api-access-vdpdw") pod "d3ae8ef1-57a9-4eef-bd6b-4122335629a2" (UID: "d3ae8ef1-57a9-4eef-bd6b-4122335629a2"). InnerVolumeSpecName "kube-api-access-vdpdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:02:32.047783 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:32.047744    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdpdw\" (UniqueName: \"kubernetes.io/projected/d3ae8ef1-57a9-4eef-bd6b-4122335629a2-kube-api-access-vdpdw\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:02:32.640254 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:32.640218    2565 generic.go:356] "Generic (PLEG): container finished" podID="d3ae8ef1-57a9-4eef-bd6b-4122335629a2" containerID="0c706fa0440fa713d9862ed4bc8b87c59bad7ec01f43c0f96fb50cc4c2b8de14" exitCode=0
Jul 16 05:02:32.640648 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:32.640268    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-nr25p"
Jul 16 05:02:32.640648 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:32.640306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-nr25p" event={"ID":"d3ae8ef1-57a9-4eef-bd6b-4122335629a2","Type":"ContainerDied","Data":"0c706fa0440fa713d9862ed4bc8b87c59bad7ec01f43c0f96fb50cc4c2b8de14"}
Jul 16 05:02:32.640648 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:32.640358    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-nr25p" event={"ID":"d3ae8ef1-57a9-4eef-bd6b-4122335629a2","Type":"ContainerDied","Data":"48cc5952dc788edd546d1790cd631ade480859b3e3d719932c0902a798243371"}
Jul 16 05:02:32.640648 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:32.640375    2565 scope.go:117] "RemoveContainer" containerID="0c706fa0440fa713d9862ed4bc8b87c59bad7ec01f43c0f96fb50cc4c2b8de14"
Jul 16 05:02:32.651129 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:32.651019    2565 scope.go:117] "RemoveContainer" containerID="0c706fa0440fa713d9862ed4bc8b87c59bad7ec01f43c0f96fb50cc4c2b8de14"
Jul 16 05:02:32.651553 ip-10-0-138-199 kubenswrapper[2565]: E0716 05:02:32.651525    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c706fa0440fa713d9862ed4bc8b87c59bad7ec01f43c0f96fb50cc4c2b8de14\": container with ID starting with 0c706fa0440fa713d9862ed4bc8b87c59bad7ec01f43c0f96fb50cc4c2b8de14 not found: ID does not exist" containerID="0c706fa0440fa713d9862ed4bc8b87c59bad7ec01f43c0f96fb50cc4c2b8de14"
Jul 16 05:02:32.651663 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:32.651565    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c706fa0440fa713d9862ed4bc8b87c59bad7ec01f43c0f96fb50cc4c2b8de14"} err="failed to get container status \"0c706fa0440fa713d9862ed4bc8b87c59bad7ec01f43c0f96fb50cc4c2b8de14\": rpc error: code = NotFound desc = could not find container \"0c706fa0440fa713d9862ed4bc8b87c59bad7ec01f43c0f96fb50cc4c2b8de14\": container with ID starting with 0c706fa0440fa713d9862ed4bc8b87c59bad7ec01f43c0f96fb50cc4c2b8de14 not found: ID does not exist"
Jul 16 05:02:32.664613 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:32.664558    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nr25p"]
Jul 16 05:02:32.666797 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:32.666763    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nr25p"]
Jul 16 05:02:33.145174 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:33.145128    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3ae8ef1-57a9-4eef-bd6b-4122335629a2" path="/var/lib/kubelet/pods/d3ae8ef1-57a9-4eef-bd6b-4122335629a2/volumes"
Jul 16 05:02:35.714766 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:35.714725    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-l9vwr"]
Jul 16 05:02:35.715250 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:35.714952    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-l9vwr" podUID="d01f4c10-70eb-4641-af56-80dd3dc04587" containerName="authorino" containerID="cri-o://e0529a8142a50c14ebc8886de3d4ee391d0911284747c55fad706f03bbd5f4d6" gracePeriod=30
Jul 16 05:02:35.974470 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:35.974404    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-l9vwr"
Jul 16 05:02:35.989390 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:35.989355    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9j6lt\" (UniqueName: \"kubernetes.io/projected/d01f4c10-70eb-4641-af56-80dd3dc04587-kube-api-access-9j6lt\") pod \"d01f4c10-70eb-4641-af56-80dd3dc04587\" (UID: \"d01f4c10-70eb-4641-af56-80dd3dc04587\") "
Jul 16 05:02:35.992652 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:35.992614    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01f4c10-70eb-4641-af56-80dd3dc04587-kube-api-access-9j6lt" (OuterVolumeSpecName: "kube-api-access-9j6lt") pod "d01f4c10-70eb-4641-af56-80dd3dc04587" (UID: "d01f4c10-70eb-4641-af56-80dd3dc04587"). InnerVolumeSpecName "kube-api-access-9j6lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:02:36.090323 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:36.090284    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j6lt\" (UniqueName: \"kubernetes.io/projected/d01f4c10-70eb-4641-af56-80dd3dc04587-kube-api-access-9j6lt\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:02:36.664225 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:36.664190    2565 generic.go:356] "Generic (PLEG): container finished" podID="d01f4c10-70eb-4641-af56-80dd3dc04587" containerID="e0529a8142a50c14ebc8886de3d4ee391d0911284747c55fad706f03bbd5f4d6" exitCode=0
Jul 16 05:02:36.664416 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:36.664237    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-l9vwr"
Jul 16 05:02:36.664416 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:36.664281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-l9vwr" event={"ID":"d01f4c10-70eb-4641-af56-80dd3dc04587","Type":"ContainerDied","Data":"e0529a8142a50c14ebc8886de3d4ee391d0911284747c55fad706f03bbd5f4d6"}
Jul 16 05:02:36.664416 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:36.664328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-l9vwr" event={"ID":"d01f4c10-70eb-4641-af56-80dd3dc04587","Type":"ContainerDied","Data":"70406672f56e30126e010312ece5c9c0b77151b345ba6aed9a7e0aced016478b"}
Jul 16 05:02:36.664416 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:36.664351    2565 scope.go:117] "RemoveContainer" containerID="e0529a8142a50c14ebc8886de3d4ee391d0911284747c55fad706f03bbd5f4d6"
Jul 16 05:02:36.674071 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:36.674051    2565 scope.go:117] "RemoveContainer" containerID="e0529a8142a50c14ebc8886de3d4ee391d0911284747c55fad706f03bbd5f4d6"
Jul 16 05:02:36.674399 ip-10-0-138-199 kubenswrapper[2565]: E0716 05:02:36.674376    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0529a8142a50c14ebc8886de3d4ee391d0911284747c55fad706f03bbd5f4d6\": container with ID starting with e0529a8142a50c14ebc8886de3d4ee391d0911284747c55fad706f03bbd5f4d6 not found: ID does not exist" containerID="e0529a8142a50c14ebc8886de3d4ee391d0911284747c55fad706f03bbd5f4d6"
Jul 16 05:02:36.674499 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:36.674407    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0529a8142a50c14ebc8886de3d4ee391d0911284747c55fad706f03bbd5f4d6"} err="failed to get container status \"e0529a8142a50c14ebc8886de3d4ee391d0911284747c55fad706f03bbd5f4d6\": rpc error: code = NotFound desc = could not find container \"e0529a8142a50c14ebc8886de3d4ee391d0911284747c55fad706f03bbd5f4d6\": container with ID starting with e0529a8142a50c14ebc8886de3d4ee391d0911284747c55fad706f03bbd5f4d6 not found: ID does not exist"
Jul 16 05:02:36.689676 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:36.689638    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-l9vwr"]
Jul 16 05:02:36.693264 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:36.693221    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-l9vwr"]
Jul 16 05:02:37.143882 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:02:37.143846    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d01f4c10-70eb-4641-af56-80dd3dc04587" path="/var/lib/kubelet/pods/d01f4c10-70eb-4641-af56-80dd3dc04587/volumes"
Jul 16 05:03:17.198697 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.198655    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-pn9n8"]
Jul 16 05:03:17.199391 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.199227    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d01f4c10-70eb-4641-af56-80dd3dc04587" containerName="authorino"
Jul 16 05:03:17.199391 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.199247    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d01f4c10-70eb-4641-af56-80dd3dc04587" containerName="authorino"
Jul 16 05:03:17.199391 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.199264    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ae8ef1-57a9-4eef-bd6b-4122335629a2" containerName="authorino"
Jul 16 05:03:17.199391 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.199273    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ae8ef1-57a9-4eef-bd6b-4122335629a2" containerName="authorino"
Jul 16 05:03:17.199391 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.199389    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3ae8ef1-57a9-4eef-bd6b-4122335629a2" containerName="authorino"
Jul 16 05:03:17.199698 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.199405    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d01f4c10-70eb-4641-af56-80dd3dc04587" containerName="authorino"
Jul 16 05:03:17.204255 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.204235    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-pn9n8"
Jul 16 05:03:17.206861 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.206835    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 16 05:03:17.207003 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.206913    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nzx72\""
Jul 16 05:03:17.207726 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.207692    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-pn9n8"]
Jul 16 05:03:17.258178 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.258136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn2sw\" (UniqueName: \"kubernetes.io/projected/f3325934-58e9-43c1-a9c3-529eca6de6fd-kube-api-access-nn2sw\") pod \"payload-processing-f9b8d9b9b-pn9n8\" (UID: \"f3325934-58e9-43c1-a9c3-529eca6de6fd\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-pn9n8"
Jul 16 05:03:17.258370 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.258200    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3325934-58e9-43c1-a9c3-529eca6de6fd-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-pn9n8\" (UID: \"f3325934-58e9-43c1-a9c3-529eca6de6fd\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-pn9n8"
Jul 16 05:03:17.281440 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.281402    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:03:17.358898 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.358861    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3325934-58e9-43c1-a9c3-529eca6de6fd-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-pn9n8\" (UID: \"f3325934-58e9-43c1-a9c3-529eca6de6fd\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-pn9n8"
Jul 16 05:03:17.359082 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.358952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn2sw\" (UniqueName: \"kubernetes.io/projected/f3325934-58e9-43c1-a9c3-529eca6de6fd-kube-api-access-nn2sw\") pod \"payload-processing-f9b8d9b9b-pn9n8\" (UID: \"f3325934-58e9-43c1-a9c3-529eca6de6fd\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-pn9n8"
Jul 16 05:03:17.359844 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.359820    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3325934-58e9-43c1-a9c3-529eca6de6fd-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-pn9n8\" (UID: \"f3325934-58e9-43c1-a9c3-529eca6de6fd\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-pn9n8"
Jul 16 05:03:17.367179 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.367152    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn2sw\" (UniqueName: \"kubernetes.io/projected/f3325934-58e9-43c1-a9c3-529eca6de6fd-kube-api-access-nn2sw\") pod \"payload-processing-f9b8d9b9b-pn9n8\" (UID: \"f3325934-58e9-43c1-a9c3-529eca6de6fd\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-pn9n8"
Jul 16 05:03:17.516134 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.516083    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-pn9n8"
Jul 16 05:03:17.659303 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.659274    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-pn9n8"]
Jul 16 05:03:17.661493 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:03:17.661433    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3325934_58e9_43c1_a9c3_529eca6de6fd.slice/crio-d5b281ae703082e88615c63739f3759072e81bc7e52fd1d4822e91eda2cac8e6 WatchSource:0}: Error finding container d5b281ae703082e88615c63739f3759072e81bc7e52fd1d4822e91eda2cac8e6: Status 404 returned error can't find the container with id d5b281ae703082e88615c63739f3759072e81bc7e52fd1d4822e91eda2cac8e6
Jul 16 05:03:17.887001 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:17.886878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-pn9n8" event={"ID":"f3325934-58e9-43c1-a9c3-529eca6de6fd","Type":"ContainerStarted","Data":"d5b281ae703082e88615c63739f3759072e81bc7e52fd1d4822e91eda2cac8e6"}
Jul 16 05:03:19.900633 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:19.900579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-pn9n8" event={"ID":"f3325934-58e9-43c1-a9c3-529eca6de6fd","Type":"ContainerStarted","Data":"892dbd048378030fe83dea6cdc87b1b36cdc17acee861e5094b4619b19d3cbdb"}
Jul 16 05:03:19.901034 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:19.900660    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-pn9n8"
Jul 16 05:03:19.920075 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:19.920021    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-pn9n8" podStartSLOduration=1.50113803 podStartE2EDuration="2.919998077s" totalImagesPullingTime="1.418860047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:03:19.917262839 +0000 UTC m=+803.333217913" watchObservedRunningTime="2026-07-16 05:03:19.919998077 +0000 UTC m=+803.335953143"
Jul 16 05:03:30.909833 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:30.909793    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-pn9n8"
Jul 16 05:03:40.050393 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:40.050351    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:03:53.134300 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:03:53.134259    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:04:04.133569 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.133528    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"]
Jul 16 05:04:04.137909 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.137881    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.141531 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.141505    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 16 05:04:04.141721 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.141558    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qtz5t\""
Jul 16 05:04:04.141721 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.141504    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 16 05:04:04.141721 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.141655    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 16 05:04:04.148007 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.147900    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"]
Jul 16 05:04:04.275843 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.275800    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9fcefb5-0081-4003-adf5-75fec2d7904d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.276035 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.275850    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9fcefb5-0081-4003-adf5-75fec2d7904d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.276035 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.275880    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9fcefb5-0081-4003-adf5-75fec2d7904d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.276035 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.275958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9fcefb5-0081-4003-adf5-75fec2d7904d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.276035 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.275993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9fcefb5-0081-4003-adf5-75fec2d7904d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.276240 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.276109    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d788f\" (UniqueName: \"kubernetes.io/projected/a9fcefb5-0081-4003-adf5-75fec2d7904d-kube-api-access-d788f\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.377177 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.377133    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9fcefb5-0081-4003-adf5-75fec2d7904d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.377177 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.377183    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9fcefb5-0081-4003-adf5-75fec2d7904d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.377457 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.377208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9fcefb5-0081-4003-adf5-75fec2d7904d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.377457 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.377224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9fcefb5-0081-4003-adf5-75fec2d7904d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.377457 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.377279    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d788f\" (UniqueName: \"kubernetes.io/projected/a9fcefb5-0081-4003-adf5-75fec2d7904d-kube-api-access-d788f\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.377457 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.377330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9fcefb5-0081-4003-adf5-75fec2d7904d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.377716 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.377636    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9fcefb5-0081-4003-adf5-75fec2d7904d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.377790 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.377734    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9fcefb5-0081-4003-adf5-75fec2d7904d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.377790 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.377748    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9fcefb5-0081-4003-adf5-75fec2d7904d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.379692 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.379669    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9fcefb5-0081-4003-adf5-75fec2d7904d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.380039 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.380021    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9fcefb5-0081-4003-adf5-75fec2d7904d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.385836 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.385768    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d788f\" (UniqueName: \"kubernetes.io/projected/a9fcefb5-0081-4003-adf5-75fec2d7904d-kube-api-access-d788f\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4bggg\" (UID: \"a9fcefb5-0081-4003-adf5-75fec2d7904d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.451007 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.450961    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:04.590827 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:04.590798    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"]
Jul 16 05:04:04.592196 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:04:04.592168    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fcefb5_0081_4003_adf5_75fec2d7904d.slice/crio-68d61f462bf52223a8fe0f08b3a23469bfab980e392a1f2aa7b16e86e2bf81c0 WatchSource:0}: Error finding container 68d61f462bf52223a8fe0f08b3a23469bfab980e392a1f2aa7b16e86e2bf81c0: Status 404 returned error can't find the container with id 68d61f462bf52223a8fe0f08b3a23469bfab980e392a1f2aa7b16e86e2bf81c0
Jul 16 05:04:05.130819 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:05.130777    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:04:05.145935 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:05.145896    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg" event={"ID":"a9fcefb5-0081-4003-adf5-75fec2d7904d","Type":"ContainerStarted","Data":"68d61f462bf52223a8fe0f08b3a23469bfab980e392a1f2aa7b16e86e2bf81c0"}
Jul 16 05:04:06.152664 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:06.152619    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg" event={"ID":"a9fcefb5-0081-4003-adf5-75fec2d7904d","Type":"ContainerStarted","Data":"892ee546992888ec68e38089bf52bdf604ec26609e85e53cb263e1ecb40b5642"}
Jul 16 05:04:06.170907 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:06.170852    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg" podStartSLOduration=1.164558411 podStartE2EDuration="2.170832165s" totalImagesPullingTime="1.006273754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:04:06.168348307 +0000 UTC m=+849.584303398" watchObservedRunningTime="2026-07-16 05:04:06.170832165 +0000 UTC m=+849.586787237"
Jul 16 05:04:14.451862 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:14.451813    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:14.452421 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:14.451890    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:14.464775 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:14.464747    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:14.939261 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:14.939226    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:04:15.211169 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:15.211083    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4bggg"
Jul 16 05:04:22.445703 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:22.445663    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:04:28.535059 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.535023    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"]
Jul 16 05:04:28.539779 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.539760    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.542709 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.542667    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 16 05:04:28.551273 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.551193    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"]
Jul 16 05:04:28.599351 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.599308    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88de6eba-4106-4646-8272-aafe80fff8a0-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.599716 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.599685    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfbqk\" (UniqueName: \"kubernetes.io/projected/88de6eba-4106-4646-8272-aafe80fff8a0-kube-api-access-cfbqk\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.599854 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.599771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88de6eba-4106-4646-8272-aafe80fff8a0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.599854 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.599839    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88de6eba-4106-4646-8272-aafe80fff8a0-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.599979 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.599881    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88de6eba-4106-4646-8272-aafe80fff8a0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.599979 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.599923    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88de6eba-4106-4646-8272-aafe80fff8a0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.700783 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.700739    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88de6eba-4106-4646-8272-aafe80fff8a0-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.700980 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.700789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88de6eba-4106-4646-8272-aafe80fff8a0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.700980 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.700826    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88de6eba-4106-4646-8272-aafe80fff8a0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.700980 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.700862    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88de6eba-4106-4646-8272-aafe80fff8a0-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.700980 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.700929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfbqk\" (UniqueName: \"kubernetes.io/projected/88de6eba-4106-4646-8272-aafe80fff8a0-kube-api-access-cfbqk\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.700980 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.700957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88de6eba-4106-4646-8272-aafe80fff8a0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.701231 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.701202    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88de6eba-4106-4646-8272-aafe80fff8a0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.701321 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.701277    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88de6eba-4106-4646-8272-aafe80fff8a0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.701397 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.701315    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88de6eba-4106-4646-8272-aafe80fff8a0-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.703273 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.703241    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88de6eba-4106-4646-8272-aafe80fff8a0-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.703532 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.703513    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88de6eba-4106-4646-8272-aafe80fff8a0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.709150 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.709117    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfbqk\" (UniqueName: \"kubernetes.io/projected/88de6eba-4106-4646-8272-aafe80fff8a0-kube-api-access-cfbqk\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp\" (UID: \"88de6eba-4106-4646-8272-aafe80fff8a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.855158 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.855056    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:28.999520 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:28.999490    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"]
Jul 16 05:04:29.001063 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:04:29.001027    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88de6eba_4106_4646_8272_aafe80fff8a0.slice/crio-965e62751586f83ce0403eb7974e50d87981635975756de563c18b87fa9d9109 WatchSource:0}: Error finding container 965e62751586f83ce0403eb7974e50d87981635975756de563c18b87fa9d9109: Status 404 returned error can't find the container with id 965e62751586f83ce0403eb7974e50d87981635975756de563c18b87fa9d9109
Jul 16 05:04:29.281103 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:29.281067    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp" event={"ID":"88de6eba-4106-4646-8272-aafe80fff8a0","Type":"ContainerStarted","Data":"965e62751586f83ce0403eb7974e50d87981635975756de563c18b87fa9d9109"}
Jul 16 05:04:30.288106 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:30.288065    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp" event={"ID":"88de6eba-4106-4646-8272-aafe80fff8a0","Type":"ContainerStarted","Data":"4b945089173d183cf7121541234cf83e6961d19e3bd7d9a3cc53e9177c7fd462"}
Jul 16 05:04:30.304486 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:30.304435    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp" podStartSLOduration=2.075639079 podStartE2EDuration="2.304419518s" totalImagesPullingTime="228.780439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:04:30.303695164 +0000 UTC m=+873.719650236" watchObservedRunningTime="2026-07-16 05:04:30.304419518 +0000 UTC m=+873.720374591"
Jul 16 05:04:31.737516 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:31.737482    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:04:38.856209 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:38.856169    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:38.856712 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:38.856233    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:38.869004 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:38.868973    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:39.348548 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:39.348518    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp"
Jul 16 05:04:57.367453 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:57.367336    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:04:57.372580 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:57.371874    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:04:57.410928 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:57.410899    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:04:57.416104 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:04:57.416078    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:05:22.720424 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:05:22.720384    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7fc69f979d-ltt5g"]
Jul 16 05:05:22.724254 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:05:22.724233    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fc69f979d-ltt5g"
Jul 16 05:05:22.727542 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:05:22.727515    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jwp4b\""
Jul 16 05:05:22.727542 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:05:22.727527    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 16 05:05:22.732496 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:05:22.732468    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7fc69f979d-ltt5g"]
Jul 16 05:05:22.804354 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:05:22.804315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnpcd\" (UniqueName: \"kubernetes.io/projected/c6c86211-b48d-46da-8ad0-3a0a28759bb9-kube-api-access-gnpcd\") pod \"authorino-7fc69f979d-ltt5g\" (UID: \"c6c86211-b48d-46da-8ad0-3a0a28759bb9\") " pod="kuadrant-system/authorino-7fc69f979d-ltt5g"
Jul 16 05:05:22.804563 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:05:22.804360    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c6c86211-b48d-46da-8ad0-3a0a28759bb9-tls-cert\") pod \"authorino-7fc69f979d-ltt5g\" (UID: \"c6c86211-b48d-46da-8ad0-3a0a28759bb9\") " pod="kuadrant-system/authorino-7fc69f979d-ltt5g"
Jul 16 05:05:22.905795 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:05:22.905758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnpcd\" (UniqueName: \"kubernetes.io/projected/c6c86211-b48d-46da-8ad0-3a0a28759bb9-kube-api-access-gnpcd\") pod \"authorino-7fc69f979d-ltt5g\" (UID: \"c6c86211-b48d-46da-8ad0-3a0a28759bb9\") " pod="kuadrant-system/authorino-7fc69f979d-ltt5g"
Jul 16 05:05:22.905795 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:05:22.905798    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c6c86211-b48d-46da-8ad0-3a0a28759bb9-tls-cert\") pod \"authorino-7fc69f979d-ltt5g\" (UID: \"c6c86211-b48d-46da-8ad0-3a0a28759bb9\") " pod="kuadrant-system/authorino-7fc69f979d-ltt5g"
Jul 16 05:05:22.908448 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:05:22.908426    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c6c86211-b48d-46da-8ad0-3a0a28759bb9-tls-cert\") pod \"authorino-7fc69f979d-ltt5g\" (UID: \"c6c86211-b48d-46da-8ad0-3a0a28759bb9\") " pod="kuadrant-system/authorino-7fc69f979d-ltt5g"
Jul 16 05:05:22.913525 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:05:22.913487    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnpcd\" (UniqueName: \"kubernetes.io/projected/c6c86211-b48d-46da-8ad0-3a0a28759bb9-kube-api-access-gnpcd\") pod \"authorino-7fc69f979d-ltt5g\" (UID: \"c6c86211-b48d-46da-8ad0-3a0a28759bb9\") " pod="kuadrant-system/authorino-7fc69f979d-ltt5g"
Jul 16 05:05:23.034906 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:05:23.034876    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fc69f979d-ltt5g"
Jul 16 05:05:23.176190 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:05:23.176159    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7fc69f979d-ltt5g"]
Jul 16 05:05:23.178460 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:05:23.178429    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6c86211_b48d_46da_8ad0_3a0a28759bb9.slice/crio-2c988b88b635deb011eb4ae13cd20e545ce627bf843b7a7477731fd7f7c57663 WatchSource:0}: Error finding container 2c988b88b635deb011eb4ae13cd20e545ce627bf843b7a7477731fd7f7c57663: Status 404 returned error can't find the container with id 2c988b88b635deb011eb4ae13cd20e545ce627bf843b7a7477731fd7f7c57663
Jul 16 05:05:23.584678 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:05:23.584637    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fc69f979d-ltt5g" event={"ID":"c6c86211-b48d-46da-8ad0-3a0a28759bb9","Type":"ContainerStarted","Data":"2c988b88b635deb011eb4ae13cd20e545ce627bf843b7a7477731fd7f7c57663"}
Jul 16 05:05:25.598458 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:05:25.598408    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fc69f979d-ltt5g" event={"ID":"c6c86211-b48d-46da-8ad0-3a0a28759bb9","Type":"ContainerStarted","Data":"9d2713e29cce2b65a323f3721c98ce28e4841116a191a02818b4a48ce07d43c5"}
Jul 16 05:05:25.616567 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:05:25.616504    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7fc69f979d-ltt5g" podStartSLOduration=1.8028414659999998 podStartE2EDuration="3.616486954s" totalImagesPullingTime="1.813645488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:05:25.615098026 +0000 UTC m=+929.031053098" watchObservedRunningTime="2026-07-16 05:05:25.616486954 +0000 UTC m=+929.032442026"
Jul 16 05:06:07.435411 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:06:07.435372    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:06:16.135604 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:06:16.135553    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:06:28.137020 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:06:28.136917    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:06:36.431795 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:06:36.431752    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:06:47.131350 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:06:47.131305    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:07:17.532012 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:07:17.531955    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:07:31.035976 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:07:31.035940    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:08:03.632545 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:08:03.632502    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:08:13.630726 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:08:13.630681    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:08:23.533701 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:08:23.533664    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:08:33.637728 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:08:33.637693    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:08:43.641049 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:08:43.640999    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:08:53.431319 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:08:53.431273    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:09:04.828541 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:09:04.828486    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:09:14.033607 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:09:14.033567    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:09:37.230230 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:09:37.230197    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:09:49.133189 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:09:49.133150    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:09:57.408697 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:09:57.408552    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:09:57.414608 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:09:57.414567    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:09:57.451184 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:09:57.451157    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:09:57.455776 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:09:57.455750    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:10:12.731394 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:10:12.731341    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:10:24.143701 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:10:24.143658    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:10:41.236552 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:10:41.236517    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:10:50.439113 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:10:50.438998    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:11:01.041092 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:11:01.041052    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:11:10.832358 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:11:10.832317    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:11:20.431340 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:11:20.431301    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:13:08.438630 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:13:08.438575    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:13:18.833904 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:13:18.833856    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:13:29.232372 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:13:29.232324    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:13:41.933408 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:13:41.933369    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:13:51.430959 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:13:51.430863    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:14:01.836742 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:14:01.836696    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:14:10.729770 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:14:10.729725    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:14:22.332495 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:14:22.332449    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:14:33.627903 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:14:33.627861    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:14:42.038450 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:14:42.038403    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:14:54.225318 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:14:54.225274    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:14:57.446870 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:14:57.446754    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:14:57.451367 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:14:57.451349    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:14:57.490426 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:14:57.490400    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:14:57.495150 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:14:57.495122    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:15:17.835844 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:15:17.835752    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:15:25.631811 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:15:25.631767    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:15:35.031143 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:15:35.031099    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:15:44.631795 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:15:44.631756    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:15:51.330535 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:15:51.330495    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:16:08.442682 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:16:08.442646    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:16:18.128128 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:16:18.128081    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:16:26.926770 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:16:26.926730    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:16:39.141755 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:16:39.141725    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:16:48.545974 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:16:48.545879    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:16:58.229430 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:16:58.229385    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:17:08.828725 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:17:08.828676    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:17:21.035708 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:17:21.035665    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:17:30.342610 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:17:30.342561    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:17:43.232642 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:17:43.232585    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:17:54.836409 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:17:54.836365    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:18:04.930672 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:18:04.930624    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:18:17.235832 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:18:17.235794    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:18:30.341371 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:18:30.341327    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:18:44.027928 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:18:44.027878    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:18:57.027352 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:18:57.027304    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:19:11.244637 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:19:11.244569    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:19:21.432609 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:19:21.432555    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:19:57.483778 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:19:57.483753    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:19:57.489855 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:19:57.488302    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:19:57.535227 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:19:57.535197    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:19:57.539929 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:19:57.539902    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:20:37.131420 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:20:37.131379    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:20:59.131522 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:20:59.131478    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:21:58.837790 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:21:58.837753    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:22:12.028390 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:22:12.028342    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:23:11.738368 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:23:11.738258    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:23:33.736345 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:23:33.736305    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:24:33.735678 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:24:33.735636    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:24:57.525441 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:24:57.525326    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:24:57.530181 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:24:57.530156    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:24:57.575241 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:24:57.575214    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:24:57.579609 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:24:57.579567    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:26:08.034521 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:26:08.034432    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:26:15.535714 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:26:15.535673    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:26:24.230170 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:26:24.230122    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:26:34.331845 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:26:34.331795    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:26:49.423821 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:26:49.423779    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:27:29.440335 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:29.440242    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:27:40.027815 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.027769    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"]
Jul 16 05:27:40.031548 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.031510    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.035383 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.035354    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-18c409ba\"/\"openshift-service-ca.crt\""
Jul 16 05:27:40.035563 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.035395    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-18c409ba\"/\"default-dockercfg-qcwpl\""
Jul 16 05:27:40.035563 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.035354    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-18c409ba\"/\"kube-root-ca.crt\""
Jul 16 05:27:40.035563 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.035354    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-18c409ba\"/\"e2e-lifecycle-model-18c409ba-kserve-self-signed-certs\""
Jul 16 05:27:40.042685 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.042655    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"]
Jul 16 05:27:40.043013 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.042993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b136c78-ff26-4a5a-ae6d-c047af196f32-tls-certs\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.043073 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.043060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-model-cache\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.043129 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.043115    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-dshm\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.043166 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.043142    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-tmp-dir\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.043256 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.043238    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-home\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.043292 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.043274    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4hsb\" (UniqueName: \"kubernetes.io/projected/2b136c78-ff26-4a5a-ae6d-c047af196f32-kube-api-access-s4hsb\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.143902 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.143855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-dshm\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.143902 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.143900    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-tmp-dir\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.144175 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.143967    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-home\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.144175 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.143986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4hsb\" (UniqueName: \"kubernetes.io/projected/2b136c78-ff26-4a5a-ae6d-c047af196f32-kube-api-access-s4hsb\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.144175 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.144027    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b136c78-ff26-4a5a-ae6d-c047af196f32-tls-certs\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.144175 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.144108    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-model-cache\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.144428 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.144403    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-tmp-dir\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.144494 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.144434    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-home\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.144494 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.144466    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-model-cache\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.146530 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.146504    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-dshm\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.146737 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.146719    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b136c78-ff26-4a5a-ae6d-c047af196f32-tls-certs\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.154421 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.154381    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4hsb\" (UniqueName: \"kubernetes.io/projected/2b136c78-ff26-4a5a-ae6d-c047af196f32-kube-api-access-s4hsb\") pod \"e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") " pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.343050 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.342937    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:40.500454 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:40.500389    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"]
Jul 16 05:27:40.501713 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:27:40.501683    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b136c78_ff26_4a5a_ae6d_c047af196f32.slice/crio-b4537975f16aeb66dc3d936ac77e8c9663414ce8b2eafdefae3cf88666894b82 WatchSource:0}: Error finding container b4537975f16aeb66dc3d936ac77e8c9663414ce8b2eafdefae3cf88666894b82: Status 404 returned error can't find the container with id b4537975f16aeb66dc3d936ac77e8c9663414ce8b2eafdefae3cf88666894b82
Jul 16 05:27:41.252488 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:41.252449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65" event={"ID":"2b136c78-ff26-4a5a-ae6d-c047af196f32","Type":"ContainerStarted","Data":"9704be93c8e1308593438346149d59aa2ae0f67d8275c293c4865ee5cb6c182d"}
Jul 16 05:27:41.252898 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:41.252496    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65" event={"ID":"2b136c78-ff26-4a5a-ae6d-c047af196f32","Type":"ContainerStarted","Data":"b4537975f16aeb66dc3d936ac77e8c9663414ce8b2eafdefae3cf88666894b82"}
Jul 16 05:27:41.286640 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:41.286553    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65" podStartSLOduration=1.286532859 podStartE2EDuration="1.286532859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:27:41.28564849 +0000 UTC m=+2264.701603564" watchObservedRunningTime="2026-07-16 05:27:41.286532859 +0000 UTC m=+2264.702487967"
Jul 16 05:27:43.227917 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:43.227865    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:27:50.343553 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:50.343493    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:50.343553 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:50.343570    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:50.357000 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:50.356967    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:27:51.333773 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:27:51.333740    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:28:06.431884 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:06.431846    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:28:27.129607 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:27.129561    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:28:32.987264 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:32.987213    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"]
Jul 16 05:28:48.009115 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.009005    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65" podUID="2b136c78-ff26-4a5a-ae6d-c047af196f32" containerName="main" containerID="cri-o://9704be93c8e1308593438346149d59aa2ae0f67d8275c293c4865ee5cb6c182d" gracePeriod=45
Jul 16 05:28:48.262868 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.262795    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:28:48.390522 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.390473    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-model-cache\") pod \"2b136c78-ff26-4a5a-ae6d-c047af196f32\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") "
Jul 16 05:28:48.390522 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.390517    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-dshm\") pod \"2b136c78-ff26-4a5a-ae6d-c047af196f32\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") "
Jul 16 05:28:48.390827 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.390567    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4hsb\" (UniqueName: \"kubernetes.io/projected/2b136c78-ff26-4a5a-ae6d-c047af196f32-kube-api-access-s4hsb\") pod \"2b136c78-ff26-4a5a-ae6d-c047af196f32\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") "
Jul 16 05:28:48.390827 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.390619    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-tmp-dir\") pod \"2b136c78-ff26-4a5a-ae6d-c047af196f32\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") "
Jul 16 05:28:48.390827 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.390641    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b136c78-ff26-4a5a-ae6d-c047af196f32-tls-certs\") pod \"2b136c78-ff26-4a5a-ae6d-c047af196f32\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") "
Jul 16 05:28:48.390827 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.390679    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-home\") pod \"2b136c78-ff26-4a5a-ae6d-c047af196f32\" (UID: \"2b136c78-ff26-4a5a-ae6d-c047af196f32\") "
Jul 16 05:28:48.391042 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.390849    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b136c78-ff26-4a5a-ae6d-c047af196f32" (UID: "2b136c78-ff26-4a5a-ae6d-c047af196f32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:28:48.391042 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.390933    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b136c78-ff26-4a5a-ae6d-c047af196f32" (UID: "2b136c78-ff26-4a5a-ae6d-c047af196f32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:28:48.391042 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.391017    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-home" (OuterVolumeSpecName: "home") pod "2b136c78-ff26-4a5a-ae6d-c047af196f32" (UID: "2b136c78-ff26-4a5a-ae6d-c047af196f32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:28:48.391042 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.391034    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-model-cache\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:28:48.393059 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.393030    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-dshm" (OuterVolumeSpecName: "dshm") pod "2b136c78-ff26-4a5a-ae6d-c047af196f32" (UID: "2b136c78-ff26-4a5a-ae6d-c047af196f32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:28:48.393187 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.393066    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b136c78-ff26-4a5a-ae6d-c047af196f32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b136c78-ff26-4a5a-ae6d-c047af196f32" (UID: "2b136c78-ff26-4a5a-ae6d-c047af196f32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 05:28:48.393187 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.393090    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b136c78-ff26-4a5a-ae6d-c047af196f32-kube-api-access-s4hsb" (OuterVolumeSpecName: "kube-api-access-s4hsb") pod "2b136c78-ff26-4a5a-ae6d-c047af196f32" (UID: "2b136c78-ff26-4a5a-ae6d-c047af196f32"). InnerVolumeSpecName "kube-api-access-s4hsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:28:48.492443 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.492384    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-dshm\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:28:48.492443 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.492435    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4hsb\" (UniqueName: \"kubernetes.io/projected/2b136c78-ff26-4a5a-ae6d-c047af196f32-kube-api-access-s4hsb\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:28:48.492443 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.492447    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-tmp-dir\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:28:48.492443 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.492458    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b136c78-ff26-4a5a-ae6d-c047af196f32-tls-certs\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:28:48.492820 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.492467    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b136c78-ff26-4a5a-ae6d-c047af196f32-home\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:28:48.644463 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.644360    2565 generic.go:356] "Generic (PLEG): container finished" podID="2b136c78-ff26-4a5a-ae6d-c047af196f32" containerID="9704be93c8e1308593438346149d59aa2ae0f67d8275c293c4865ee5cb6c182d" exitCode=0
Jul 16 05:28:48.644463 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.644407    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"
Jul 16 05:28:48.644463 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.644448    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65" event={"ID":"2b136c78-ff26-4a5a-ae6d-c047af196f32","Type":"ContainerDied","Data":"9704be93c8e1308593438346149d59aa2ae0f67d8275c293c4865ee5cb6c182d"}
Jul 16 05:28:48.644764 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.644495    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65" event={"ID":"2b136c78-ff26-4a5a-ae6d-c047af196f32","Type":"ContainerDied","Data":"b4537975f16aeb66dc3d936ac77e8c9663414ce8b2eafdefae3cf88666894b82"}
Jul 16 05:28:48.644764 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.644516    2565 scope.go:117] "RemoveContainer" containerID="9704be93c8e1308593438346149d59aa2ae0f67d8275c293c4865ee5cb6c182d"
Jul 16 05:28:48.654317 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.654295    2565 scope.go:117] "RemoveContainer" containerID="9704be93c8e1308593438346149d59aa2ae0f67d8275c293c4865ee5cb6c182d"
Jul 16 05:28:48.654641 ip-10-0-138-199 kubenswrapper[2565]: E0716 05:28:48.654617    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9704be93c8e1308593438346149d59aa2ae0f67d8275c293c4865ee5cb6c182d\": container with ID starting with 9704be93c8e1308593438346149d59aa2ae0f67d8275c293c4865ee5cb6c182d not found: ID does not exist" containerID="9704be93c8e1308593438346149d59aa2ae0f67d8275c293c4865ee5cb6c182d"
Jul 16 05:28:48.654715 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.654651    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9704be93c8e1308593438346149d59aa2ae0f67d8275c293c4865ee5cb6c182d"} err="failed to get container status \"9704be93c8e1308593438346149d59aa2ae0f67d8275c293c4865ee5cb6c182d\": rpc error: code = NotFound desc = could not find container \"9704be93c8e1308593438346149d59aa2ae0f67d8275c293c4865ee5cb6c182d\": container with ID starting with 9704be93c8e1308593438346149d59aa2ae0f67d8275c293c4865ee5cb6c182d not found: ID does not exist"
Jul 16 05:28:48.665294 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.665263    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"]
Jul 16 05:28:48.670672 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:48.670643    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-18c409ba/e2e-lifecycle-model-18c409ba-kserve-bdb97bbdc-j5v65"]
Jul 16 05:28:49.142329 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:28:49.142293    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b136c78-ff26-4a5a-ae6d-c047af196f32" path="/var/lib/kubelet/pods/2b136c78-ff26-4a5a-ae6d-c047af196f32/volumes"
Jul 16 05:29:44.435243 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:29:44.435204    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:29:57.566210 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:29:57.566097    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:29:57.570719 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:29:57.570692    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:29:57.613787 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:29:57.613758    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:29:57.618624 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:29:57.618580    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:29:58.335373 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:29:58.335336    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:30:46.537759 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:46.537714    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:30:53.028481 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.028438    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"]
Jul 16 05:30:53.029309 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.028845    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b136c78-ff26-4a5a-ae6d-c047af196f32" containerName="main"
Jul 16 05:30:53.029309 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.028857    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b136c78-ff26-4a5a-ae6d-c047af196f32" containerName="main"
Jul 16 05:30:53.029309 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.028932    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b136c78-ff26-4a5a-ae6d-c047af196f32" containerName="main"
Jul 16 05:30:53.032187 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.032164    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.034810 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.034782    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a956af\"/\"default-dockercfg-75l9x\""
Jul 16 05:30:53.035798 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.035776    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a956af\"/\"kube-root-ca.crt\""
Jul 16 05:30:53.036069 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.036017    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a956af\"/\"openshift-service-ca.crt\""
Jul 16 05:30:53.036212 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.036112    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a956af\"/\"test-model-a956af-kserve-self-signed-certs\""
Jul 16 05:30:53.039819 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.039786    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"]
Jul 16 05:30:53.109421 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.109384    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-tmp-dir\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.109634 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.109469    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-dshm\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.109634 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.109524    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-model-cache\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.109634 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.109578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v8mr\" (UniqueName: \"kubernetes.io/projected/29a2b561-482f-4149-879d-3e4d94820257-kube-api-access-2v8mr\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.109766 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.109655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a2b561-482f-4149-879d-3e4d94820257-tls-certs\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.109766 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.109683    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-home\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.210177 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.210135    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-model-cache\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.210177 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.210175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v8mr\" (UniqueName: \"kubernetes.io/projected/29a2b561-482f-4149-879d-3e4d94820257-kube-api-access-2v8mr\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.210456 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.210196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a2b561-482f-4149-879d-3e4d94820257-tls-certs\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.210456 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.210230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-home\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.210456 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.210286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-tmp-dir\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.210456 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.210358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-dshm\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.210814 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.210779    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-tmp-dir\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.210814 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.210803    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-model-cache\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.210987 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.210926    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-home\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.213006 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.212982    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-dshm\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.213242 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.213224    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a2b561-482f-4149-879d-3e4d94820257-tls-certs\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.217990 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.217956    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v8mr\" (UniqueName: \"kubernetes.io/projected/29a2b561-482f-4149-879d-3e4d94820257-kube-api-access-2v8mr\") pod \"test-model-a956af-kserve-79b4955d9-gvb84\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") " pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.344528 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.344417    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:30:53.499096 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:53.499062    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"]
Jul 16 05:30:53.501198 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:30:53.501159    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29a2b561_482f_4149_879d_3e4d94820257.slice/crio-90273fd2bb12a01c7c185dcb3c5a46798f6ccfc8ad9c165a6b6331d77c92a617 WatchSource:0}: Error finding container 90273fd2bb12a01c7c185dcb3c5a46798f6ccfc8ad9c165a6b6331d77c92a617: Status 404 returned error can't find the container with id 90273fd2bb12a01c7c185dcb3c5a46798f6ccfc8ad9c165a6b6331d77c92a617
Jul 16 05:30:54.380708 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:54.380660    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84" event={"ID":"29a2b561-482f-4149-879d-3e4d94820257","Type":"ContainerStarted","Data":"f9b8f41a2fb7be499d44bd2b76878f816048fa09099753adb2bd8d2d19327a38"}
Jul 16 05:30:54.381199 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:54.380720    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84" event={"ID":"29a2b561-482f-4149-879d-3e4d94820257","Type":"ContainerStarted","Data":"90273fd2bb12a01c7c185dcb3c5a46798f6ccfc8ad9c165a6b6331d77c92a617"}
Jul 16 05:30:54.398302 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:54.398236    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84" podStartSLOduration=1.3982147569999999 podStartE2EDuration="1.398214757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:30:54.395522474 +0000 UTC m=+2457.811477548" watchObservedRunningTime="2026-07-16 05:30:54.398214757 +0000 UTC m=+2457.814169829"
Jul 16 05:30:59.526571 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:30:59.526518    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:31:03.345373 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:03.345327    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:31:03.345883 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:03.345410    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:31:03.358373 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:03.358341    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:31:03.448908 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:03.448876    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:31:25.740835 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.740793    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"]
Jul 16 05:31:25.744814 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.744783    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.748279 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.748251    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-309054\"/\"default-dockercfg-xbd8r\""
Jul 16 05:31:25.748699 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.748251    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-309054\"/\"kube-root-ca.crt\""
Jul 16 05:31:25.748845 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.748365    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-309054\"/\"openshift-service-ca.crt\""
Jul 16 05:31:25.748845 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.748447    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-309054\"/\"test-model-309054-kserve-self-signed-certs\""
Jul 16 05:31:25.752221 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.752189    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"]
Jul 16 05:31:25.807792 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.807752    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-home\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.808155 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.808108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-dshm\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.808319 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.808167    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-tmp-dir\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.808319 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.808252    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-model-cache\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.808319 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.808297    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d04bd45-fb0f-4057-91e8-92320cd3279a-tls-certs\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.808490 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.808346    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x89t8\" (UniqueName: \"kubernetes.io/projected/4d04bd45-fb0f-4057-91e8-92320cd3279a-kube-api-access-x89t8\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.908998 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.908954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-dshm\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.908998 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.909009    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-tmp-dir\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.909283 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.909047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-model-cache\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.909283 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.909073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d04bd45-fb0f-4057-91e8-92320cd3279a-tls-certs\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.909283 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.909103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x89t8\" (UniqueName: \"kubernetes.io/projected/4d04bd45-fb0f-4057-91e8-92320cd3279a-kube-api-access-x89t8\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.909283 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.909154    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-home\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.909535 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.909512    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-tmp-dir\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.909627 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.909552    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-model-cache\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.909627 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.909581    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-home\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.911558 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.911538    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-dshm\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.911753 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.911737    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d04bd45-fb0f-4057-91e8-92320cd3279a-tls-certs\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:25.917480 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:25.917452    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x89t8\" (UniqueName: \"kubernetes.io/projected/4d04bd45-fb0f-4057-91e8-92320cd3279a-kube-api-access-x89t8\") pod \"test-model-309054-kserve-c547bff55-lksm9\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") " pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:26.056101 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:26.055984    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:26.205464 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:26.205421    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"]
Jul 16 05:31:26.207489 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:31:26.207451    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d04bd45_fb0f_4057_91e8_92320cd3279a.slice/crio-db467ebf422fd8ce23c4dd187f372b669f3b6f57dddb4b0925b0e5dc3f1ce6bc WatchSource:0}: Error finding container db467ebf422fd8ce23c4dd187f372b669f3b6f57dddb4b0925b0e5dc3f1ce6bc: Status 404 returned error can't find the container with id db467ebf422fd8ce23c4dd187f372b669f3b6f57dddb4b0925b0e5dc3f1ce6bc
Jul 16 05:31:26.571505 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:26.571472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9" event={"ID":"4d04bd45-fb0f-4057-91e8-92320cd3279a","Type":"ContainerStarted","Data":"f32bb1ce89897271bfc8849609f699f263b752a963bd2c15a79819987051f120"}
Jul 16 05:31:26.571718 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:26.571514    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9" event={"ID":"4d04bd45-fb0f-4057-91e8-92320cd3279a","Type":"ContainerStarted","Data":"db467ebf422fd8ce23c4dd187f372b669f3b6f57dddb4b0925b0e5dc3f1ce6bc"}
Jul 16 05:31:26.589019 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:26.588961    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9" podStartSLOduration=1.588941014 podStartE2EDuration="1.588941014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:31:26.586944505 +0000 UTC m=+2490.002899579" watchObservedRunningTime="2026-07-16 05:31:26.588941014 +0000 UTC m=+2490.004896087"
Jul 16 05:31:30.130172 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:30.130124    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:31:36.056822 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:36.056763    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:36.057396 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:36.056845    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:36.070012 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:36.069982    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:36.652880 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:36.652842    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:31:44.592307 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:44.592264    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"]
Jul 16 05:31:45.556164 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:45.556125    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"]
Jul 16 05:31:49.034117 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:49.034014    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:31:59.614648 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.614604    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84" podUID="29a2b561-482f-4149-879d-3e4d94820257" containerName="main" containerID="cri-o://f9b8f41a2fb7be499d44bd2b76878f816048fa09099753adb2bd8d2d19327a38" gracePeriod=45
Jul 16 05:31:59.776010 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.775969    2565 generic.go:356] "Generic (PLEG): container finished" podID="29a2b561-482f-4149-879d-3e4d94820257" containerID="f9b8f41a2fb7be499d44bd2b76878f816048fa09099753adb2bd8d2d19327a38" exitCode=0
Jul 16 05:31:59.776203 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.776049    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84" event={"ID":"29a2b561-482f-4149-879d-3e4d94820257","Type":"ContainerDied","Data":"f9b8f41a2fb7be499d44bd2b76878f816048fa09099753adb2bd8d2d19327a38"}
Jul 16 05:31:59.875257 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.875188    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:31:59.931089 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.931043    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v8mr\" (UniqueName: \"kubernetes.io/projected/29a2b561-482f-4149-879d-3e4d94820257-kube-api-access-2v8mr\") pod \"29a2b561-482f-4149-879d-3e4d94820257\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") "
Jul 16 05:31:59.931283 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.931110    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-home\") pod \"29a2b561-482f-4149-879d-3e4d94820257\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") "
Jul 16 05:31:59.931283 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.931136    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-model-cache\") pod \"29a2b561-482f-4149-879d-3e4d94820257\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") "
Jul 16 05:31:59.931283 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.931190    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-tmp-dir\") pod \"29a2b561-482f-4149-879d-3e4d94820257\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") "
Jul 16 05:31:59.931283 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.931218    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a2b561-482f-4149-879d-3e4d94820257-tls-certs\") pod \"29a2b561-482f-4149-879d-3e4d94820257\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") "
Jul 16 05:31:59.931283 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.931246    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-dshm\") pod \"29a2b561-482f-4149-879d-3e4d94820257\" (UID: \"29a2b561-482f-4149-879d-3e4d94820257\") "
Jul 16 05:31:59.931577 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.931426    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-home" (OuterVolumeSpecName: "home") pod "29a2b561-482f-4149-879d-3e4d94820257" (UID: "29a2b561-482f-4149-879d-3e4d94820257"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:31:59.931577 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.931436    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29a2b561-482f-4149-879d-3e4d94820257" (UID: "29a2b561-482f-4149-879d-3e4d94820257"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:31:59.931577 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.931545    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-model-cache" (OuterVolumeSpecName: "model-cache") pod "29a2b561-482f-4149-879d-3e4d94820257" (UID: "29a2b561-482f-4149-879d-3e4d94820257"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:31:59.931756 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.931691    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-home\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:31:59.931756 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.931705    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-model-cache\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:31:59.931756 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.931715    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-tmp-dir\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:31:59.933834 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.933798    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a2b561-482f-4149-879d-3e4d94820257-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29a2b561-482f-4149-879d-3e4d94820257" (UID: "29a2b561-482f-4149-879d-3e4d94820257"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 05:31:59.933834 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.933821    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a2b561-482f-4149-879d-3e4d94820257-kube-api-access-2v8mr" (OuterVolumeSpecName: "kube-api-access-2v8mr") pod "29a2b561-482f-4149-879d-3e4d94820257" (UID: "29a2b561-482f-4149-879d-3e4d94820257"). InnerVolumeSpecName "kube-api-access-2v8mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:31:59.933992 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:31:59.933818    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-dshm" (OuterVolumeSpecName: "dshm") pod "29a2b561-482f-4149-879d-3e4d94820257" (UID: "29a2b561-482f-4149-879d-3e4d94820257"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:32:00.033142 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.033097    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v8mr\" (UniqueName: \"kubernetes.io/projected/29a2b561-482f-4149-879d-3e4d94820257-kube-api-access-2v8mr\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:32:00.033142 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.033132    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a2b561-482f-4149-879d-3e4d94820257-tls-certs\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:32:00.033142 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.033145    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a2b561-482f-4149-879d-3e4d94820257-dshm\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:32:00.578981 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.578930    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9" podUID="4d04bd45-fb0f-4057-91e8-92320cd3279a" containerName="main" containerID="cri-o://f32bb1ce89897271bfc8849609f699f263b752a963bd2c15a79819987051f120" gracePeriod=45
Jul 16 05:32:00.786116 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.786073    2565 generic.go:356] "Generic (PLEG): container finished" podID="4d04bd45-fb0f-4057-91e8-92320cd3279a" containerID="f32bb1ce89897271bfc8849609f699f263b752a963bd2c15a79819987051f120" exitCode=0
Jul 16 05:32:00.786579 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.786117    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9" event={"ID":"4d04bd45-fb0f-4057-91e8-92320cd3279a","Type":"ContainerDied","Data":"f32bb1ce89897271bfc8849609f699f263b752a963bd2c15a79819987051f120"}
Jul 16 05:32:00.788112 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.788072    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"
Jul 16 05:32:00.788112 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.788083    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84" event={"ID":"29a2b561-482f-4149-879d-3e4d94820257","Type":"ContainerDied","Data":"90273fd2bb12a01c7c185dcb3c5a46798f6ccfc8ad9c165a6b6331d77c92a617"}
Jul 16 05:32:00.788350 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.788152    2565 scope.go:117] "RemoveContainer" containerID="f9b8f41a2fb7be499d44bd2b76878f816048fa09099753adb2bd8d2d19327a38"
Jul 16 05:32:00.812045 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.812010    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"]
Jul 16 05:32:00.814445 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.814419    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a956af/test-model-a956af-kserve-79b4955d9-gvb84"]
Jul 16 05:32:00.849691 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.849655    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:32:00.941965 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.941921    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-dshm\") pod \"4d04bd45-fb0f-4057-91e8-92320cd3279a\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") "
Jul 16 05:32:00.942144 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.941997    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d04bd45-fb0f-4057-91e8-92320cd3279a-tls-certs\") pod \"4d04bd45-fb0f-4057-91e8-92320cd3279a\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") "
Jul 16 05:32:00.942144 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.942036    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x89t8\" (UniqueName: \"kubernetes.io/projected/4d04bd45-fb0f-4057-91e8-92320cd3279a-kube-api-access-x89t8\") pod \"4d04bd45-fb0f-4057-91e8-92320cd3279a\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") "
Jul 16 05:32:00.942228 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.942144    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-tmp-dir\") pod \"4d04bd45-fb0f-4057-91e8-92320cd3279a\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") "
Jul 16 05:32:00.942228 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.942181    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-model-cache\") pod \"4d04bd45-fb0f-4057-91e8-92320cd3279a\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") "
Jul 16 05:32:00.942325 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.942243    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-home\") pod \"4d04bd45-fb0f-4057-91e8-92320cd3279a\" (UID: \"4d04bd45-fb0f-4057-91e8-92320cd3279a\") "
Jul 16 05:32:00.942449 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.942425    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d04bd45-fb0f-4057-91e8-92320cd3279a" (UID: "4d04bd45-fb0f-4057-91e8-92320cd3279a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:32:00.942517 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.942477    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d04bd45-fb0f-4057-91e8-92320cd3279a" (UID: "4d04bd45-fb0f-4057-91e8-92320cd3279a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:32:00.942573 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.942546    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-home" (OuterVolumeSpecName: "home") pod "4d04bd45-fb0f-4057-91e8-92320cd3279a" (UID: "4d04bd45-fb0f-4057-91e8-92320cd3279a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:32:00.942676 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.942632    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-tmp-dir\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:32:00.942676 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.942655    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-model-cache\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:32:00.944603 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.944562    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-dshm" (OuterVolumeSpecName: "dshm") pod "4d04bd45-fb0f-4057-91e8-92320cd3279a" (UID: "4d04bd45-fb0f-4057-91e8-92320cd3279a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:32:00.944738 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.944566    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d04bd45-fb0f-4057-91e8-92320cd3279a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d04bd45-fb0f-4057-91e8-92320cd3279a" (UID: "4d04bd45-fb0f-4057-91e8-92320cd3279a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 05:32:00.944738 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:00.944629    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d04bd45-fb0f-4057-91e8-92320cd3279a-kube-api-access-x89t8" (OuterVolumeSpecName: "kube-api-access-x89t8") pod "4d04bd45-fb0f-4057-91e8-92320cd3279a" (UID: "4d04bd45-fb0f-4057-91e8-92320cd3279a"). InnerVolumeSpecName "kube-api-access-x89t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:32:01.043098 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:01.043053    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-dshm\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:32:01.043098 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:01.043089    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d04bd45-fb0f-4057-91e8-92320cd3279a-tls-certs\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:32:01.043098 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:01.043101    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x89t8\" (UniqueName: \"kubernetes.io/projected/4d04bd45-fb0f-4057-91e8-92320cd3279a-kube-api-access-x89t8\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:32:01.043098 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:01.043109    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d04bd45-fb0f-4057-91e8-92320cd3279a-home\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:32:01.143604 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:01.143489    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29a2b561-482f-4149-879d-3e4d94820257" path="/var/lib/kubelet/pods/29a2b561-482f-4149-879d-3e4d94820257/volumes"
Jul 16 05:32:01.796166 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:01.796132    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9" event={"ID":"4d04bd45-fb0f-4057-91e8-92320cd3279a","Type":"ContainerDied","Data":"db467ebf422fd8ce23c4dd187f372b669f3b6f57dddb4b0925b0e5dc3f1ce6bc"}
Jul 16 05:32:01.796166 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:01.796155    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"
Jul 16 05:32:01.796678 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:01.796184    2565 scope.go:117] "RemoveContainer" containerID="f32bb1ce89897271bfc8849609f699f263b752a963bd2c15a79819987051f120"
Jul 16 05:32:01.813400 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:01.813359    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"]
Jul 16 05:32:01.818035 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:01.818000    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-309054/test-model-309054-kserve-c547bff55-lksm9"]
Jul 16 05:32:03.142849 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:03.142806    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d04bd45-fb0f-4057-91e8-92320cd3279a" path="/var/lib/kubelet/pods/4d04bd45-fb0f-4057-91e8-92320cd3279a/volumes"
Jul 16 05:32:07.035921 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:07.035879    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:32:25.937698 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:25.937651    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:32:46.334635 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:46.334567    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:32:58.879211 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:32:58.879169    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:33:15.333541 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:33:15.333505    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:33:42.432184 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:33:42.432137    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:34:02.927149 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:02.927097    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:34:17.938148 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:17.938102    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:34:21.336899 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.336859    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"]
Jul 16 05:34:21.337409 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.337350    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a2b561-482f-4149-879d-3e4d94820257" containerName="main"
Jul 16 05:34:21.337409 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.337366    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a2b561-482f-4149-879d-3e4d94820257" containerName="main"
Jul 16 05:34:21.337409 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.337376    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d04bd45-fb0f-4057-91e8-92320cd3279a" containerName="main"
Jul 16 05:34:21.337409 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.337382    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d04bd45-fb0f-4057-91e8-92320cd3279a" containerName="main"
Jul 16 05:34:21.338031 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.337440    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a2b561-482f-4149-879d-3e4d94820257" containerName="main"
Jul 16 05:34:21.338031 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.337448    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d04bd45-fb0f-4057-91e8-92320cd3279a" containerName="main"
Jul 16 05:34:21.340956 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.340926    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.344839 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.344808    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-615edb\"/\"openshift-service-ca.crt\""
Jul 16 05:34:21.345662 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.344808    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-615edb\"/\"auth-test-model-615edb-kserve-self-signed-certs\""
Jul 16 05:34:21.345909 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.344811    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-615edb\"/\"kube-root-ca.crt\""
Jul 16 05:34:21.346021 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.344836    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-615edb\"/\"default-dockercfg-f9wpk\""
Jul 16 05:34:21.348988 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.348961    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"]
Jul 16 05:34:21.473738 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.473698    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-home\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.473738 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.473738    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0912ae73-64f7-4efc-be61-3d8e5d193e0c-tls-certs\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.473964 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.473766    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-dshm\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.473964 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.473826    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-tmp-dir\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.473964 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.473906    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-model-cache\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.473964 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.473960    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsrkk\" (UniqueName: \"kubernetes.io/projected/0912ae73-64f7-4efc-be61-3d8e5d193e0c-kube-api-access-gsrkk\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.575258 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.575203    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsrkk\" (UniqueName: \"kubernetes.io/projected/0912ae73-64f7-4efc-be61-3d8e5d193e0c-kube-api-access-gsrkk\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.575258 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.575270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-home\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.575502 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.575296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0912ae73-64f7-4efc-be61-3d8e5d193e0c-tls-certs\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.575502 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.575331    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-dshm\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.575502 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.575358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-tmp-dir\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.575502 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.575419    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-model-cache\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.575783 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.575758    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-home\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.575832 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.575806    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-tmp-dir\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.575870 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.575837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-model-cache\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.578015 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.577982    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-dshm\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.578254 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.578233    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0912ae73-64f7-4efc-be61-3d8e5d193e0c-tls-certs\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.583316 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.583287    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsrkk\" (UniqueName: \"kubernetes.io/projected/0912ae73-64f7-4efc-be61-3d8e5d193e0c-kube-api-access-gsrkk\") pod \"auth-test-model-615edb-kserve-747685d5bc-69b6f\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") " pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.653910 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.653798    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:21.811235 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:21.811199    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"]
Jul 16 05:34:21.812847 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:34:21.812813    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0912ae73_64f7_4efc_be61_3d8e5d193e0c.slice/crio-635d2a623df47abf5ebd73e1792376e8437358ee22f68f29b60eb3b530be611d WatchSource:0}: Error finding container 635d2a623df47abf5ebd73e1792376e8437358ee22f68f29b60eb3b530be611d: Status 404 returned error can't find the container with id 635d2a623df47abf5ebd73e1792376e8437358ee22f68f29b60eb3b530be611d
Jul 16 05:34:22.607511 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:22.607475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f" event={"ID":"0912ae73-64f7-4efc-be61-3d8e5d193e0c","Type":"ContainerStarted","Data":"0de204bf94622045a099f633f5f9de69d7e163fa7999a6cbb1d8b64b159aae56"}
Jul 16 05:34:22.607932 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:22.607519    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f" event={"ID":"0912ae73-64f7-4efc-be61-3d8e5d193e0c","Type":"ContainerStarted","Data":"635d2a623df47abf5ebd73e1792376e8437358ee22f68f29b60eb3b530be611d"}
Jul 16 05:34:22.623083 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:22.623011    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f" podStartSLOduration=1.622993472 podStartE2EDuration="1.622993472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:34:22.622173619 +0000 UTC m=+2666.038128694" watchObservedRunningTime="2026-07-16 05:34:22.622993472 +0000 UTC m=+2666.038948545"
Jul 16 05:34:25.534581 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:25.534540    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:34:31.654617 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:31.654486    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:31.654617 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:31.654552    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:31.667890 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:31.667858    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:32.678197 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:32.678166    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:34:45.636088 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.636045    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"]
Jul 16 05:34:45.640295 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.640269    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.642964 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.642935    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e36375\"/\"kube-root-ca.crt\""
Jul 16 05:34:45.643936 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.643905    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e36375\"/\"default-dockercfg-9lmh4\""
Jul 16 05:34:45.643936 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.643905    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e36375\"/\"openshift-service-ca.crt\""
Jul 16 05:34:45.644161 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.643965    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e36375\"/\"auth-test-model-e36375-kserve-self-signed-certs\""
Jul 16 05:34:45.648154 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.648126    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"]
Jul 16 05:34:45.710583 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.710537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-home\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.710813 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.710620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-dshm\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.710813 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.710673    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc8c6df-4448-401b-9a36-60430c5484f0-tls-certs\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.710813 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.710717    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-tmp-dir\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.710813 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.710737    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tlm4\" (UniqueName: \"kubernetes.io/projected/8dc8c6df-4448-401b-9a36-60430c5484f0-kube-api-access-5tlm4\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.710813 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.710764    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-model-cache\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.811364 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.811321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-home\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.811554 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.811378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-dshm\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.811554 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.811412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc8c6df-4448-401b-9a36-60430c5484f0-tls-certs\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.811554 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.811448    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-tmp-dir\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.811786 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.811623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tlm4\" (UniqueName: \"kubernetes.io/projected/8dc8c6df-4448-401b-9a36-60430c5484f0-kube-api-access-5tlm4\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.811786 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.811693    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-model-cache\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.811877 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.811824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-home\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.811877 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.811836    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-tmp-dir\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.812057 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.812035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-model-cache\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.814074 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.814042    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-dshm\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.814244 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.814227    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc8c6df-4448-401b-9a36-60430c5484f0-tls-certs\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.819930 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.819898    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tlm4\" (UniqueName: \"kubernetes.io/projected/8dc8c6df-4448-401b-9a36-60430c5484f0-kube-api-access-5tlm4\") pod \"auth-test-model-e36375-kserve-7b85bf64b7-t2hgk\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") " pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:45.952787 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:45.952688    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:46.106811 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:46.106778    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"]
Jul 16 05:34:46.109959 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:34:46.109927    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dc8c6df_4448_401b_9a36_60430c5484f0.slice/crio-9f091602a3071b77f5440be141ae505343d5d98ec6ed085aea503f696b658917 WatchSource:0}: Error finding container 9f091602a3071b77f5440be141ae505343d5d98ec6ed085aea503f696b658917: Status 404 returned error can't find the container with id 9f091602a3071b77f5440be141ae505343d5d98ec6ed085aea503f696b658917
Jul 16 05:34:46.746914 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:46.746866    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk" event={"ID":"8dc8c6df-4448-401b-9a36-60430c5484f0","Type":"ContainerStarted","Data":"f3759e12121ff4dc508eba5260521eeda74dbc5816dd3768d23a59cb58d6a277"}
Jul 16 05:34:46.747325 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:46.746926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk" event={"ID":"8dc8c6df-4448-401b-9a36-60430c5484f0","Type":"ContainerStarted","Data":"9f091602a3071b77f5440be141ae505343d5d98ec6ed085aea503f696b658917"}
Jul 16 05:34:46.764039 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:46.763981    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk" podStartSLOduration=1.7639604389999999 podStartE2EDuration="1.763960439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:34:46.761928013 +0000 UTC m=+2690.177883089" watchObservedRunningTime="2026-07-16 05:34:46.763960439 +0000 UTC m=+2690.179915521"
Jul 16 05:34:49.231923 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:49.231823    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:34:55.953168 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:55.953111    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:55.953691 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:55.953208    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:55.966051 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:55.966021    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:56.817180 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:56.817145    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:34:57.604848 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:57.604726    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:34:57.609649 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:57.609625    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:34:57.654696 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:57.654669    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:34:57.659688 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:34:57.659656    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:35:11.142999 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:35:11.142963    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:35:28.530739 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:35:28.530697    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:35:45.940312 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:35:45.940271    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:36:03.231321 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:03.231280    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:36:21.636398 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:21.636312    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:36:38.833970 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:38.833918    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:36:50.269655 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:50.269579    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"]
Jul 16 05:36:51.126605 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:51.126544    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"]
Jul 16 05:36:53.946291 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:53.946260    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"]
Jul 16 05:36:53.949963 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:53.949941    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:53.952805 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:53.952781    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-615edb\"/\"sub-test-model-615edb-kserve-self-signed-certs\""
Jul 16 05:36:53.956978 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:53.956951    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"]
Jul 16 05:36:54.035309 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.035272    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d149414e-3dc9-4d1b-aade-e6d96a866b0a-tls-certs\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.035547 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.035331    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-home\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.035547 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.035414    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-tmp-dir\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.035547 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.035473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-dshm\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.035547 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.035501    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgd75\" (UniqueName: \"kubernetes.io/projected/d149414e-3dc9-4d1b-aade-e6d96a866b0a-kube-api-access-cgd75\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.035547 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.035537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-model-cache\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.136669 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.136628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-dshm\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.136669 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.136672    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgd75\" (UniqueName: \"kubernetes.io/projected/d149414e-3dc9-4d1b-aade-e6d96a866b0a-kube-api-access-cgd75\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.136895 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.136694    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-model-cache\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.136895 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.136770    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d149414e-3dc9-4d1b-aade-e6d96a866b0a-tls-certs\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.136895 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.136809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-home\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.136895 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.136865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-tmp-dir\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.137225 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.137199    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-model-cache\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.137296 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.137255    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-home\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.137343 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.137331    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-tmp-dir\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.139266 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.139243    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-dshm\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.139462 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.139445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d149414e-3dc9-4d1b-aade-e6d96a866b0a-tls-certs\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.144810 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.144780    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgd75\" (UniqueName: \"kubernetes.io/projected/d149414e-3dc9-4d1b-aade-e6d96a866b0a-kube-api-access-cgd75\") pod \"sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") " pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.261561 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.261504    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:36:54.410264 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.410231    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"]
Jul 16 05:36:54.415777 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:36:54.415744    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd149414e_3dc9_4d1b_aade_e6d96a866b0a.slice/crio-14ba8c4f96ef923f157574c0a382f07896ee97e9d88b45a265a19a8ea0ac94f5 WatchSource:0}: Error finding container 14ba8c4f96ef923f157574c0a382f07896ee97e9d88b45a265a19a8ea0ac94f5: Status 404 returned error can't find the container with id 14ba8c4f96ef923f157574c0a382f07896ee97e9d88b45a265a19a8ea0ac94f5
Jul 16 05:36:54.502866 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.502826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc" event={"ID":"d149414e-3dc9-4d1b-aade-e6d96a866b0a","Type":"ContainerStarted","Data":"1b9af0128ec8aa4ead24e86c9eb6fb6cc371254a4441e17550f4dcf278808079"}
Jul 16 05:36:54.503074 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.502875    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc" event={"ID":"d149414e-3dc9-4d1b-aade-e6d96a866b0a","Type":"ContainerStarted","Data":"14ba8c4f96ef923f157574c0a382f07896ee97e9d88b45a265a19a8ea0ac94f5"}
Jul 16 05:36:54.518325 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:54.518214    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc" podStartSLOduration=1.518194477 podStartE2EDuration="1.518194477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:36:54.517159909 +0000 UTC m=+2817.933114995" watchObservedRunningTime="2026-07-16 05:36:54.518194477 +0000 UTC m=+2817.934149565"
Jul 16 05:36:57.336238 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:36:57.336194    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:37:04.262545 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:04.262485    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:37:04.263014 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:04.262575    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:37:04.276008 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:04.275969    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:37:04.573152 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:04.573060    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:37:05.293807 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.293762    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f" podUID="0912ae73-64f7-4efc-be61-3d8e5d193e0c" containerName="main" containerID="cri-o://0de204bf94622045a099f633f5f9de69d7e163fa7999a6cbb1d8b64b159aae56" gracePeriod=45
Jul 16 05:37:05.557536 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.557455    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:37:05.568322 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.568288    2565 generic.go:356] "Generic (PLEG): container finished" podID="0912ae73-64f7-4efc-be61-3d8e5d193e0c" containerID="0de204bf94622045a099f633f5f9de69d7e163fa7999a6cbb1d8b64b159aae56" exitCode=0
Jul 16 05:37:05.568322 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.568326    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"
Jul 16 05:37:05.568547 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.568368    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f" event={"ID":"0912ae73-64f7-4efc-be61-3d8e5d193e0c","Type":"ContainerDied","Data":"0de204bf94622045a099f633f5f9de69d7e163fa7999a6cbb1d8b64b159aae56"}
Jul 16 05:37:05.568547 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.568416    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f" event={"ID":"0912ae73-64f7-4efc-be61-3d8e5d193e0c","Type":"ContainerDied","Data":"635d2a623df47abf5ebd73e1792376e8437358ee22f68f29b60eb3b530be611d"}
Jul 16 05:37:05.568547 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.568440    2565 scope.go:117] "RemoveContainer" containerID="0de204bf94622045a099f633f5f9de69d7e163fa7999a6cbb1d8b64b159aae56"
Jul 16 05:37:05.580022 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.580000    2565 scope.go:117] "RemoveContainer" containerID="0de204bf94622045a099f633f5f9de69d7e163fa7999a6cbb1d8b64b159aae56"
Jul 16 05:37:05.580432 ip-10-0-138-199 kubenswrapper[2565]: E0716 05:37:05.580405    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de204bf94622045a099f633f5f9de69d7e163fa7999a6cbb1d8b64b159aae56\": container with ID starting with 0de204bf94622045a099f633f5f9de69d7e163fa7999a6cbb1d8b64b159aae56 not found: ID does not exist" containerID="0de204bf94622045a099f633f5f9de69d7e163fa7999a6cbb1d8b64b159aae56"
Jul 16 05:37:05.580552 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.580440    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de204bf94622045a099f633f5f9de69d7e163fa7999a6cbb1d8b64b159aae56"} err="failed to get container status \"0de204bf94622045a099f633f5f9de69d7e163fa7999a6cbb1d8b64b159aae56\": rpc error: code = NotFound desc = could not find container \"0de204bf94622045a099f633f5f9de69d7e163fa7999a6cbb1d8b64b159aae56\": container with ID starting with 0de204bf94622045a099f633f5f9de69d7e163fa7999a6cbb1d8b64b159aae56 not found: ID does not exist"
Jul 16 05:37:05.649666 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.649627    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-dshm\") pod \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") "
Jul 16 05:37:05.649838 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.649692    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsrkk\" (UniqueName: \"kubernetes.io/projected/0912ae73-64f7-4efc-be61-3d8e5d193e0c-kube-api-access-gsrkk\") pod \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") "
Jul 16 05:37:05.649838 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.649721    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0912ae73-64f7-4efc-be61-3d8e5d193e0c-tls-certs\") pod \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") "
Jul 16 05:37:05.649838 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.649739    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-model-cache\") pod \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") "
Jul 16 05:37:05.649838 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.649794    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-tmp-dir\") pod \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") "
Jul 16 05:37:05.650049 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.649890    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-home\") pod \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\" (UID: \"0912ae73-64f7-4efc-be61-3d8e5d193e0c\") "
Jul 16 05:37:05.650049 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.649987    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0912ae73-64f7-4efc-be61-3d8e5d193e0c" (UID: "0912ae73-64f7-4efc-be61-3d8e5d193e0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:37:05.650157 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.650082    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0912ae73-64f7-4efc-be61-3d8e5d193e0c" (UID: "0912ae73-64f7-4efc-be61-3d8e5d193e0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:37:05.650236 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.650202    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-home" (OuterVolumeSpecName: "home") pod "0912ae73-64f7-4efc-be61-3d8e5d193e0c" (UID: "0912ae73-64f7-4efc-be61-3d8e5d193e0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:37:05.650358 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.650323    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-home\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:37:05.650358 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.650345    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-model-cache\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:37:05.650454 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.650361    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-tmp-dir\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:37:05.652033 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.651998    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-dshm" (OuterVolumeSpecName: "dshm") pod "0912ae73-64f7-4efc-be61-3d8e5d193e0c" (UID: "0912ae73-64f7-4efc-be61-3d8e5d193e0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:37:05.652166 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.652105    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0912ae73-64f7-4efc-be61-3d8e5d193e0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0912ae73-64f7-4efc-be61-3d8e5d193e0c" (UID: "0912ae73-64f7-4efc-be61-3d8e5d193e0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 05:37:05.652166 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.652142    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0912ae73-64f7-4efc-be61-3d8e5d193e0c-kube-api-access-gsrkk" (OuterVolumeSpecName: "kube-api-access-gsrkk") pod "0912ae73-64f7-4efc-be61-3d8e5d193e0c" (UID: "0912ae73-64f7-4efc-be61-3d8e5d193e0c"). InnerVolumeSpecName "kube-api-access-gsrkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:37:05.751376 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.751324    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0912ae73-64f7-4efc-be61-3d8e5d193e0c-dshm\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:37:05.751376 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.751364    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsrkk\" (UniqueName: \"kubernetes.io/projected/0912ae73-64f7-4efc-be61-3d8e5d193e0c-kube-api-access-gsrkk\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:37:05.751376 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.751381    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0912ae73-64f7-4efc-be61-3d8e5d193e0c-tls-certs\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:37:05.890773 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.890741    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"]
Jul 16 05:37:05.894385 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:05.894349    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-615edb/auth-test-model-615edb-kserve-747685d5bc-69b6f"]
Jul 16 05:37:06.149544 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.149428    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk" podUID="8dc8c6df-4448-401b-9a36-60430c5484f0" containerName="main" containerID="cri-o://f3759e12121ff4dc508eba5260521eeda74dbc5816dd3768d23a59cb58d6a277" gracePeriod=45
Jul 16 05:37:06.423025 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.422987    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:37:06.456887 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.456842    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-home\") pod \"8dc8c6df-4448-401b-9a36-60430c5484f0\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") "
Jul 16 05:37:06.457094 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.456925    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc8c6df-4448-401b-9a36-60430c5484f0-tls-certs\") pod \"8dc8c6df-4448-401b-9a36-60430c5484f0\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") "
Jul 16 05:37:06.457094 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.457000    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tlm4\" (UniqueName: \"kubernetes.io/projected/8dc8c6df-4448-401b-9a36-60430c5484f0-kube-api-access-5tlm4\") pod \"8dc8c6df-4448-401b-9a36-60430c5484f0\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") "
Jul 16 05:37:06.457094 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.457065    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-tmp-dir\") pod \"8dc8c6df-4448-401b-9a36-60430c5484f0\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") "
Jul 16 05:37:06.457270 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.457179    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-home" (OuterVolumeSpecName: "home") pod "8dc8c6df-4448-401b-9a36-60430c5484f0" (UID: "8dc8c6df-4448-401b-9a36-60430c5484f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:37:06.457270 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.457188    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-dshm\") pod \"8dc8c6df-4448-401b-9a36-60430c5484f0\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") "
Jul 16 05:37:06.457270 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.457257    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-model-cache\") pod \"8dc8c6df-4448-401b-9a36-60430c5484f0\" (UID: \"8dc8c6df-4448-401b-9a36-60430c5484f0\") "
Jul 16 05:37:06.457408 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.457357    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8dc8c6df-4448-401b-9a36-60430c5484f0" (UID: "8dc8c6df-4448-401b-9a36-60430c5484f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:37:06.457503 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.457481    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "8dc8c6df-4448-401b-9a36-60430c5484f0" (UID: "8dc8c6df-4448-401b-9a36-60430c5484f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:37:06.457691 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.457665    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-tmp-dir\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:37:06.457758 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.457690    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-model-cache\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:37:06.457758 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.457704    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-home\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:37:06.459523 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.459498    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dc8c6df-4448-401b-9a36-60430c5484f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8dc8c6df-4448-401b-9a36-60430c5484f0" (UID: "8dc8c6df-4448-401b-9a36-60430c5484f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 05:37:06.459523 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.459505    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-dshm" (OuterVolumeSpecName: "dshm") pod "8dc8c6df-4448-401b-9a36-60430c5484f0" (UID: "8dc8c6df-4448-401b-9a36-60430c5484f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:37:06.459728 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.459711    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc8c6df-4448-401b-9a36-60430c5484f0-kube-api-access-5tlm4" (OuterVolumeSpecName: "kube-api-access-5tlm4") pod "8dc8c6df-4448-401b-9a36-60430c5484f0" (UID: "8dc8c6df-4448-401b-9a36-60430c5484f0"). InnerVolumeSpecName "kube-api-access-5tlm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:37:06.559182 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.559142    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tlm4\" (UniqueName: \"kubernetes.io/projected/8dc8c6df-4448-401b-9a36-60430c5484f0-kube-api-access-5tlm4\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:37:06.559182 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.559179    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dc8c6df-4448-401b-9a36-60430c5484f0-dshm\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:37:06.559182 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.559191    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc8c6df-4448-401b-9a36-60430c5484f0-tls-certs\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:37:06.578121 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.578084    2565 generic.go:356] "Generic (PLEG): container finished" podID="8dc8c6df-4448-401b-9a36-60430c5484f0" containerID="f3759e12121ff4dc508eba5260521eeda74dbc5816dd3768d23a59cb58d6a277" exitCode=0
Jul 16 05:37:06.578327 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.578131    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk" event={"ID":"8dc8c6df-4448-401b-9a36-60430c5484f0","Type":"ContainerDied","Data":"f3759e12121ff4dc508eba5260521eeda74dbc5816dd3768d23a59cb58d6a277"}
Jul 16 05:37:06.578327 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.578144    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"
Jul 16 05:37:06.578327 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.578164    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk" event={"ID":"8dc8c6df-4448-401b-9a36-60430c5484f0","Type":"ContainerDied","Data":"9f091602a3071b77f5440be141ae505343d5d98ec6ed085aea503f696b658917"}
Jul 16 05:37:06.578327 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.578182    2565 scope.go:117] "RemoveContainer" containerID="f3759e12121ff4dc508eba5260521eeda74dbc5816dd3768d23a59cb58d6a277"
Jul 16 05:37:06.589414 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.589389    2565 scope.go:117] "RemoveContainer" containerID="f3759e12121ff4dc508eba5260521eeda74dbc5816dd3768d23a59cb58d6a277"
Jul 16 05:37:06.589796 ip-10-0-138-199 kubenswrapper[2565]: E0716 05:37:06.589770    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3759e12121ff4dc508eba5260521eeda74dbc5816dd3768d23a59cb58d6a277\": container with ID starting with f3759e12121ff4dc508eba5260521eeda74dbc5816dd3768d23a59cb58d6a277 not found: ID does not exist" containerID="f3759e12121ff4dc508eba5260521eeda74dbc5816dd3768d23a59cb58d6a277"
Jul 16 05:37:06.589866 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.589811    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3759e12121ff4dc508eba5260521eeda74dbc5816dd3768d23a59cb58d6a277"} err="failed to get container status \"f3759e12121ff4dc508eba5260521eeda74dbc5816dd3768d23a59cb58d6a277\": rpc error: code = NotFound desc = could not find container \"f3759e12121ff4dc508eba5260521eeda74dbc5816dd3768d23a59cb58d6a277\": container with ID starting with f3759e12121ff4dc508eba5260521eeda74dbc5816dd3768d23a59cb58d6a277 not found: ID does not exist"
Jul 16 05:37:06.602146 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.602100    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"]
Jul 16 05:37:06.605418 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:06.605380    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e36375/auth-test-model-e36375-kserve-7b85bf64b7-t2hgk"]
Jul 16 05:37:06.621486 ip-10-0-138-199 kubenswrapper[2565]: E0716 05:37:06.621452    2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dc8c6df_4448_401b_9a36_60430c5484f0.slice\": RecentStats: unable to find data in memory cache]"
Jul 16 05:37:07.148532 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:07.148488    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0912ae73-64f7-4efc-be61-3d8e5d193e0c" path="/var/lib/kubelet/pods/0912ae73-64f7-4efc-be61-3d8e5d193e0c/volumes"
Jul 16 05:37:07.148937 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:07.148921    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dc8c6df-4448-401b-9a36-60430c5484f0" path="/var/lib/kubelet/pods/8dc8c6df-4448-401b-9a36-60430c5484f0/volumes"
Jul 16 05:37:15.542743 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.542702    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"]
Jul 16 05:37:15.543270 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.543132    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0912ae73-64f7-4efc-be61-3d8e5d193e0c" containerName="main"
Jul 16 05:37:15.543270 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.543147    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0912ae73-64f7-4efc-be61-3d8e5d193e0c" containerName="main"
Jul 16 05:37:15.543270 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.543162    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dc8c6df-4448-401b-9a36-60430c5484f0" containerName="main"
Jul 16 05:37:15.543270 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.543168    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dc8c6df-4448-401b-9a36-60430c5484f0" containerName="main"
Jul 16 05:37:15.543270 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.543235    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0912ae73-64f7-4efc-be61-3d8e5d193e0c" containerName="main"
Jul 16 05:37:15.543270 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.543246    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dc8c6df-4448-401b-9a36-60430c5484f0" containerName="main"
Jul 16 05:37:15.547768 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.547740    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.551543 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.551512    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e36375\"/\"sub-test-model-e36375-kserve-self-signed-certs\""
Jul 16 05:37:15.551734 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.551512    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e36375\"/\"default-dockercfg-9lmh4\""
Jul 16 05:37:15.551734 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.551512    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e36375\"/\"kube-root-ca.crt\""
Jul 16 05:37:15.551734 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.551512    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e36375\"/\"openshift-service-ca.crt\""
Jul 16 05:37:15.556238 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.555992    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"]
Jul 16 05:37:15.648652 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.648568    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-model-cache\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.648862 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.648682    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-tmp-dir\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.648862 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.648738    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8718d58a-effe-4643-b5e3-482703ca6b01-tls-certs\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.648862 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.648793    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-home\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.648862 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.648810    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-968xx\" (UniqueName: \"kubernetes.io/projected/8718d58a-effe-4643-b5e3-482703ca6b01-kube-api-access-968xx\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.649020 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.648902    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-dshm\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.749660 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.749582    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-model-cache\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.749875 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.749681    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-tmp-dir\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.749875 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.749716    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8718d58a-effe-4643-b5e3-482703ca6b01-tls-certs\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.749875 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.749777    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-home\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.749875 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.749803    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-968xx\" (UniqueName: \"kubernetes.io/projected/8718d58a-effe-4643-b5e3-482703ca6b01-kube-api-access-968xx\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.749875 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.749854    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-dshm\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.750162 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.750035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-model-cache\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.750162 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.750075    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-tmp-dir\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.750162 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.750123    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-home\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.752374 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.752348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-dshm\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.752576 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.752555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8718d58a-effe-4643-b5e3-482703ca6b01-tls-certs\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.757987 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.757956    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-968xx\" (UniqueName: \"kubernetes.io/projected/8718d58a-effe-4643-b5e3-482703ca6b01-kube-api-access-968xx\") pod \"sub-test-model-e36375-kserve-5996bb6885-4kfqm\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") " pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:15.859576 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:15.859468    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:16.009375 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:16.009343    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"]
Jul 16 05:37:16.012386 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:37:16.012355    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8718d58a_effe_4643_b5e3_482703ca6b01.slice/crio-3d97f4a743e3ffe292b6ea5632fcd64672fa3eb11437c4884babd7adcc35aa40 WatchSource:0}: Error finding container 3d97f4a743e3ffe292b6ea5632fcd64672fa3eb11437c4884babd7adcc35aa40: Status 404 returned error can't find the container with id 3d97f4a743e3ffe292b6ea5632fcd64672fa3eb11437c4884babd7adcc35aa40
Jul 16 05:37:16.642784 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:16.642748    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm" event={"ID":"8718d58a-effe-4643-b5e3-482703ca6b01","Type":"ContainerStarted","Data":"10e796708f3079fea70c03e687e600753484410e1ce20a50f8b4b5fd87497f7f"}
Jul 16 05:37:16.643198 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:16.642801    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm" event={"ID":"8718d58a-effe-4643-b5e3-482703ca6b01","Type":"ContainerStarted","Data":"3d97f4a743e3ffe292b6ea5632fcd64672fa3eb11437c4884babd7adcc35aa40"}
Jul 16 05:37:16.659699 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:16.659640    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm" podStartSLOduration=1.65962115 podStartE2EDuration="1.65962115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:37:16.657954198 +0000 UTC m=+2840.073909272" watchObservedRunningTime="2026-07-16 05:37:16.65962115 +0000 UTC m=+2840.075576220"
Jul 16 05:37:20.731531 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:20.731487    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:37:25.860030 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:25.859985    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:25.860465 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:25.860057    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:25.873116 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:25.873085    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:26.711652 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:26.711619    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:37:57.729139 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:37:57.729046    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:38:17.332454 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:17.332408    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:38:23.264294 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:23.264245    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"]
Jul 16 05:38:24.145374 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:24.145328    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"]
Jul 16 05:38:28.739022 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.738980    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"]
Jul 16 05:38:28.742866 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.742836    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.745653 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.745619    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-615edb\"/\"rate-test-model-615edb-kserve-self-signed-certs\""
Jul 16 05:38:28.751370 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.751338    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"]
Jul 16 05:38:28.849322 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.849277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-dshm\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.849322 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.849323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/256d1ed7-4b8d-4791-a33f-b24df47937a3-tls-certs\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.849573 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.849355    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zspqb\" (UniqueName: \"kubernetes.io/projected/256d1ed7-4b8d-4791-a33f-b24df47937a3-kube-api-access-zspqb\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.849573 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.849374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-tmp-dir\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.849573 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.849490    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-home\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.849573 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.849527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-model-cache\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.950748 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.950703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-home\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.950748 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.950748    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-model-cache\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.951003 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.950816    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-dshm\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.951003 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.950847    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/256d1ed7-4b8d-4791-a33f-b24df47937a3-tls-certs\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.951003 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.950886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zspqb\" (UniqueName: \"kubernetes.io/projected/256d1ed7-4b8d-4791-a33f-b24df47937a3-kube-api-access-zspqb\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.951003 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.950916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-tmp-dir\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.951207 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.951185    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-home\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.951279 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.951238    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-model-cache\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.951355 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.951312    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-tmp-dir\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.953439 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.953417    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-dshm\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.953619 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.953578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/256d1ed7-4b8d-4791-a33f-b24df47937a3-tls-certs\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:28.959176 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:28.959140    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zspqb\" (UniqueName: \"kubernetes.io/projected/256d1ed7-4b8d-4791-a33f-b24df47937a3-kube-api-access-zspqb\") pod \"rate-test-model-615edb-kserve-756587567f-2kq8z\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") " pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:29.054084 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:29.053983    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:29.209904 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:29.209871    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"]
Jul 16 05:38:29.211898 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:38:29.211868    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod256d1ed7_4b8d_4791_a33f_b24df47937a3.slice/crio-150472ea1cc91e8017440232b7e8555bf944f025b31ab456bbd8560f0fee8cb0 WatchSource:0}: Error finding container 150472ea1cc91e8017440232b7e8555bf944f025b31ab456bbd8560f0fee8cb0: Status 404 returned error can't find the container with id 150472ea1cc91e8017440232b7e8555bf944f025b31ab456bbd8560f0fee8cb0
Jul 16 05:38:30.075068 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:30.075021    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z" event={"ID":"256d1ed7-4b8d-4791-a33f-b24df47937a3","Type":"ContainerStarted","Data":"39d58e4952d31282e9960be5d3c882a552f59711095cff00a83da4983abb9915"}
Jul 16 05:38:30.075519 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:30.075079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z" event={"ID":"256d1ed7-4b8d-4791-a33f-b24df47937a3","Type":"ContainerStarted","Data":"150472ea1cc91e8017440232b7e8555bf944f025b31ab456bbd8560f0fee8cb0"}
Jul 16 05:38:30.093890 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:30.093726    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z" podStartSLOduration=2.093705114 podStartE2EDuration="2.093705114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:38:30.093320761 +0000 UTC m=+2913.509275845" watchObservedRunningTime="2026-07-16 05:38:30.093705114 +0000 UTC m=+2913.509660187"
Jul 16 05:38:35.835750 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:35.835702    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:38:38.286497 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.286439    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc" podUID="d149414e-3dc9-4d1b-aade-e6d96a866b0a" containerName="main" containerID="cri-o://1b9af0128ec8aa4ead24e86c9eb6fb6cc371254a4441e17550f4dcf278808079" gracePeriod=45
Jul 16 05:38:38.559273 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.559240    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:38:38.651257 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.651214    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgd75\" (UniqueName: \"kubernetes.io/projected/d149414e-3dc9-4d1b-aade-e6d96a866b0a-kube-api-access-cgd75\") pod \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") "
Jul 16 05:38:38.651456 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.651272    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-dshm\") pod \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") "
Jul 16 05:38:38.651456 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.651347    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-home\") pod \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") "
Jul 16 05:38:38.651456 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.651369    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-tmp-dir\") pod \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") "
Jul 16 05:38:38.651456 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.651393    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-model-cache\") pod \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") "
Jul 16 05:38:38.651726 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.651462    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d149414e-3dc9-4d1b-aade-e6d96a866b0a-tls-certs\") pod \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\" (UID: \"d149414e-3dc9-4d1b-aade-e6d96a866b0a\") "
Jul 16 05:38:38.651726 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.651691    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d149414e-3dc9-4d1b-aade-e6d96a866b0a" (UID: "d149414e-3dc9-4d1b-aade-e6d96a866b0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:38:38.651726 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.651706    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-home" (OuterVolumeSpecName: "home") pod "d149414e-3dc9-4d1b-aade-e6d96a866b0a" (UID: "d149414e-3dc9-4d1b-aade-e6d96a866b0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:38:38.651885 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.651735    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d149414e-3dc9-4d1b-aade-e6d96a866b0a" (UID: "d149414e-3dc9-4d1b-aade-e6d96a866b0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:38:38.651942 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.651918    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-home\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:38:38.651942 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.651933    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-tmp-dir\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:38:38.652020 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.651943    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-model-cache\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:38:38.653780 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.653756    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-dshm" (OuterVolumeSpecName: "dshm") pod "d149414e-3dc9-4d1b-aade-e6d96a866b0a" (UID: "d149414e-3dc9-4d1b-aade-e6d96a866b0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:38:38.654302 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.654280    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d149414e-3dc9-4d1b-aade-e6d96a866b0a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d149414e-3dc9-4d1b-aade-e6d96a866b0a" (UID: "d149414e-3dc9-4d1b-aade-e6d96a866b0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 05:38:38.654396 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.654321    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d149414e-3dc9-4d1b-aade-e6d96a866b0a-kube-api-access-cgd75" (OuterVolumeSpecName: "kube-api-access-cgd75") pod "d149414e-3dc9-4d1b-aade-e6d96a866b0a" (UID: "d149414e-3dc9-4d1b-aade-e6d96a866b0a"). InnerVolumeSpecName "kube-api-access-cgd75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:38:38.753168 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.753124    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgd75\" (UniqueName: \"kubernetes.io/projected/d149414e-3dc9-4d1b-aade-e6d96a866b0a-kube-api-access-cgd75\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:38:38.753168 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.753159    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d149414e-3dc9-4d1b-aade-e6d96a866b0a-dshm\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:38:38.753168 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:38.753170    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d149414e-3dc9-4d1b-aade-e6d96a866b0a-tls-certs\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:38:39.054702 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.054659    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:39.054896 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.054722    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:39.073370 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.073329    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:39.134074 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.134034    2565 generic.go:356] "Generic (PLEG): container finished" podID="d149414e-3dc9-4d1b-aade-e6d96a866b0a" containerID="1b9af0128ec8aa4ead24e86c9eb6fb6cc371254a4441e17550f4dcf278808079" exitCode=0
Jul 16 05:38:39.134263 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.134101    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"
Jul 16 05:38:39.134263 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.134130    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc" event={"ID":"d149414e-3dc9-4d1b-aade-e6d96a866b0a","Type":"ContainerDied","Data":"1b9af0128ec8aa4ead24e86c9eb6fb6cc371254a4441e17550f4dcf278808079"}
Jul 16 05:38:39.134263 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.134180    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc" event={"ID":"d149414e-3dc9-4d1b-aade-e6d96a866b0a","Type":"ContainerDied","Data":"14ba8c4f96ef923f157574c0a382f07896ee97e9d88b45a265a19a8ea0ac94f5"}
Jul 16 05:38:39.134263 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.134204    2565 scope.go:117] "RemoveContainer" containerID="1b9af0128ec8aa4ead24e86c9eb6fb6cc371254a4441e17550f4dcf278808079"
Jul 16 05:38:39.146105 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.146083    2565 scope.go:117] "RemoveContainer" containerID="1b9af0128ec8aa4ead24e86c9eb6fb6cc371254a4441e17550f4dcf278808079"
Jul 16 05:38:39.146473 ip-10-0-138-199 kubenswrapper[2565]: E0716 05:38:39.146452    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9af0128ec8aa4ead24e86c9eb6fb6cc371254a4441e17550f4dcf278808079\": container with ID starting with 1b9af0128ec8aa4ead24e86c9eb6fb6cc371254a4441e17550f4dcf278808079 not found: ID does not exist" containerID="1b9af0128ec8aa4ead24e86c9eb6fb6cc371254a4441e17550f4dcf278808079"
Jul 16 05:38:39.146539 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.146486    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9af0128ec8aa4ead24e86c9eb6fb6cc371254a4441e17550f4dcf278808079"} err="failed to get container status \"1b9af0128ec8aa4ead24e86c9eb6fb6cc371254a4441e17550f4dcf278808079\": rpc error: code = NotFound desc = could not find container \"1b9af0128ec8aa4ead24e86c9eb6fb6cc371254a4441e17550f4dcf278808079\": container with ID starting with 1b9af0128ec8aa4ead24e86c9eb6fb6cc371254a4441e17550f4dcf278808079 not found: ID does not exist"
Jul 16 05:38:39.148041 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.148020    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:38:39.156481 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.156444    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"]
Jul 16 05:38:39.160838 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.160804    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-615edb/sub-test-model-615edb-kserve-6d7bf4d46f-tc2pc"]
Jul 16 05:38:39.169433 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.169397    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm" podUID="8718d58a-effe-4643-b5e3-482703ca6b01" containerName="main" containerID="cri-o://10e796708f3079fea70c03e687e600753484410e1ce20a50f8b4b5fd87497f7f" gracePeriod=45
Jul 16 05:38:39.442902 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.442866    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:38:39.561422 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.561380    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-tmp-dir\") pod \"8718d58a-effe-4643-b5e3-482703ca6b01\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") "
Jul 16 05:38:39.561645 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.561445    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8718d58a-effe-4643-b5e3-482703ca6b01-tls-certs\") pod \"8718d58a-effe-4643-b5e3-482703ca6b01\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") "
Jul 16 05:38:39.561645 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.561523    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-dshm\") pod \"8718d58a-effe-4643-b5e3-482703ca6b01\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") "
Jul 16 05:38:39.561766 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.561642    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-home\") pod \"8718d58a-effe-4643-b5e3-482703ca6b01\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") "
Jul 16 05:38:39.561766 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.561682    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-model-cache\") pod \"8718d58a-effe-4643-b5e3-482703ca6b01\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") "
Jul 16 05:38:39.561766 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.561748    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-968xx\" (UniqueName: \"kubernetes.io/projected/8718d58a-effe-4643-b5e3-482703ca6b01-kube-api-access-968xx\") pod \"8718d58a-effe-4643-b5e3-482703ca6b01\" (UID: \"8718d58a-effe-4643-b5e3-482703ca6b01\") "
Jul 16 05:38:39.561925 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.561764    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8718d58a-effe-4643-b5e3-482703ca6b01" (UID: "8718d58a-effe-4643-b5e3-482703ca6b01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:38:39.561982 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.561938    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-home" (OuterVolumeSpecName: "home") pod "8718d58a-effe-4643-b5e3-482703ca6b01" (UID: "8718d58a-effe-4643-b5e3-482703ca6b01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:38:39.562026 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.561975    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-model-cache" (OuterVolumeSpecName: "model-cache") pod "8718d58a-effe-4643-b5e3-482703ca6b01" (UID: "8718d58a-effe-4643-b5e3-482703ca6b01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:38:39.562070 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.562049    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-tmp-dir\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:38:39.562070 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.562062    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-home\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:38:39.562070 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.562070    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-model-cache\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:38:39.564173 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.564139    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8718d58a-effe-4643-b5e3-482703ca6b01-kube-api-access-968xx" (OuterVolumeSpecName: "kube-api-access-968xx") pod "8718d58a-effe-4643-b5e3-482703ca6b01" (UID: "8718d58a-effe-4643-b5e3-482703ca6b01"). InnerVolumeSpecName "kube-api-access-968xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:38:39.564452 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.564430    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8718d58a-effe-4643-b5e3-482703ca6b01-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8718d58a-effe-4643-b5e3-482703ca6b01" (UID: "8718d58a-effe-4643-b5e3-482703ca6b01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 05:38:39.564452 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.564439    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-dshm" (OuterVolumeSpecName: "dshm") pod "8718d58a-effe-4643-b5e3-482703ca6b01" (UID: "8718d58a-effe-4643-b5e3-482703ca6b01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:38:39.663015 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.662920    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-968xx\" (UniqueName: \"kubernetes.io/projected/8718d58a-effe-4643-b5e3-482703ca6b01-kube-api-access-968xx\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:38:39.663015 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.662953    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8718d58a-effe-4643-b5e3-482703ca6b01-tls-certs\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:38:39.663015 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:39.662968    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8718d58a-effe-4643-b5e3-482703ca6b01-dshm\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:38:40.143449 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:40.143412    2565 generic.go:356] "Generic (PLEG): container finished" podID="8718d58a-effe-4643-b5e3-482703ca6b01" containerID="10e796708f3079fea70c03e687e600753484410e1ce20a50f8b4b5fd87497f7f" exitCode=0
Jul 16 05:38:40.143685 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:40.143461    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"
Jul 16 05:38:40.143685 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:40.143495    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm" event={"ID":"8718d58a-effe-4643-b5e3-482703ca6b01","Type":"ContainerDied","Data":"10e796708f3079fea70c03e687e600753484410e1ce20a50f8b4b5fd87497f7f"}
Jul 16 05:38:40.143685 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:40.143545    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm" event={"ID":"8718d58a-effe-4643-b5e3-482703ca6b01","Type":"ContainerDied","Data":"3d97f4a743e3ffe292b6ea5632fcd64672fa3eb11437c4884babd7adcc35aa40"}
Jul 16 05:38:40.143685 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:40.143570    2565 scope.go:117] "RemoveContainer" containerID="10e796708f3079fea70c03e687e600753484410e1ce20a50f8b4b5fd87497f7f"
Jul 16 05:38:40.154243 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:40.154222    2565 scope.go:117] "RemoveContainer" containerID="10e796708f3079fea70c03e687e600753484410e1ce20a50f8b4b5fd87497f7f"
Jul 16 05:38:40.154689 ip-10-0-138-199 kubenswrapper[2565]: E0716 05:38:40.154666    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e796708f3079fea70c03e687e600753484410e1ce20a50f8b4b5fd87497f7f\": container with ID starting with 10e796708f3079fea70c03e687e600753484410e1ce20a50f8b4b5fd87497f7f not found: ID does not exist" containerID="10e796708f3079fea70c03e687e600753484410e1ce20a50f8b4b5fd87497f7f"
Jul 16 05:38:40.154763 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:40.154700    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e796708f3079fea70c03e687e600753484410e1ce20a50f8b4b5fd87497f7f"} err="failed to get container status \"10e796708f3079fea70c03e687e600753484410e1ce20a50f8b4b5fd87497f7f\": rpc error: code = NotFound desc = could not find container \"10e796708f3079fea70c03e687e600753484410e1ce20a50f8b4b5fd87497f7f\": container with ID starting with 10e796708f3079fea70c03e687e600753484410e1ce20a50f8b4b5fd87497f7f not found: ID does not exist"
Jul 16 05:38:40.164900 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:40.164860    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"]
Jul 16 05:38:40.168467 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:40.168425    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e36375/sub-test-model-e36375-kserve-5996bb6885-4kfqm"]
Jul 16 05:38:41.142755 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:41.142719    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8718d58a-effe-4643-b5e3-482703ca6b01" path="/var/lib/kubelet/pods/8718d58a-effe-4643-b5e3-482703ca6b01/volumes"
Jul 16 05:38:41.143161 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:41.143146    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d149414e-3dc9-4d1b-aade-e6d96a866b0a" path="/var/lib/kubelet/pods/d149414e-3dc9-4d1b-aade-e6d96a866b0a/volumes"
Jul 16 05:38:50.938498 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:50.938457    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"]
Jul 16 05:38:50.940204 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:50.940172    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d149414e-3dc9-4d1b-aade-e6d96a866b0a" containerName="main"
Jul 16 05:38:50.940408 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:50.940395    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d149414e-3dc9-4d1b-aade-e6d96a866b0a" containerName="main"
Jul 16 05:38:50.940509 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:50.940500    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8718d58a-effe-4643-b5e3-482703ca6b01" containerName="main"
Jul 16 05:38:50.940573 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:50.940565    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8718d58a-effe-4643-b5e3-482703ca6b01" containerName="main"
Jul 16 05:38:50.940909 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:50.940892    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8718d58a-effe-4643-b5e3-482703ca6b01" containerName="main"
Jul 16 05:38:50.941022 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:50.941012    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d149414e-3dc9-4d1b-aade-e6d96a866b0a" containerName="main"
Jul 16 05:38:50.946682 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:50.946645    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:50.949518 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:50.949474    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e36375\"/\"kube-root-ca.crt\""
Jul 16 05:38:50.949725 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:50.949522    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"]
Jul 16 05:38:50.950461 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:50.950418    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e36375\"/\"openshift-service-ca.crt\""
Jul 16 05:38:50.950642 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:50.950574    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e36375\"/\"default-dockercfg-9lmh4\""
Jul 16 05:38:50.950721 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:50.950672    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e36375\"/\"rate-test-model-e36375-kserve-self-signed-certs\""
Jul 16 05:38:51.071468 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.071423    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-dshm\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.071715 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.071477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-home\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.071715 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.071568    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-tls-certs\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.071715 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.071652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-model-cache\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.071715 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.071697    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btcwl\" (UniqueName: \"kubernetes.io/projected/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-kube-api-access-btcwl\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.071715 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.071718    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-tmp-dir\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.172283 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.172245    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btcwl\" (UniqueName: \"kubernetes.io/projected/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-kube-api-access-btcwl\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.172283 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.172287    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-tmp-dir\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.172554 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.172352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-dshm\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.172554 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.172372    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-home\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.172554 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.172401    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-tls-certs\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.172554 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.172435    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-model-cache\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.172904 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.172873    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-tmp-dir\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.172962 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.172906    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-home\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.172962 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.172922    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-model-cache\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.175075 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.175048    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-dshm\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.175202 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.175184    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-tls-certs\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.180725 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.180690    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btcwl\" (UniqueName: \"kubernetes.io/projected/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-kube-api-access-btcwl\") pod \"rate-test-model-e36375-kserve-6f68f444ff-7lhb5\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") " pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.258142 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.258101    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:38:51.411217 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:51.411187    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"]
Jul 16 05:38:51.412860 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:38:51.412829    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad0f5ffc_0978_4ae4_899f_a93ddb17b9a6.slice/crio-d7986f5776ba78485a44c05b25c51d7867e6bd220458680b368a23cc0cc5a941 WatchSource:0}: Error finding container d7986f5776ba78485a44c05b25c51d7867e6bd220458680b368a23cc0cc5a941: Status 404 returned error can't find the container with id d7986f5776ba78485a44c05b25c51d7867e6bd220458680b368a23cc0cc5a941
Jul 16 05:38:52.217917 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:52.217873    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5" event={"ID":"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6","Type":"ContainerStarted","Data":"0e8bc9a4d88c5c94e5096189c02def51d06829276128520520f0f73d81124f2e"}
Jul 16 05:38:52.217917 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:52.217925    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5" event={"ID":"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6","Type":"ContainerStarted","Data":"d7986f5776ba78485a44c05b25c51d7867e6bd220458680b368a23cc0cc5a941"}
Jul 16 05:38:52.234646 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:52.234561    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5" podStartSLOduration=2.234540073 podStartE2EDuration="2.234540073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:38:52.233027808 +0000 UTC m=+2935.648982881" watchObservedRunningTime="2026-07-16 05:38:52.234540073 +0000 UTC m=+2935.650495147"
Jul 16 05:38:55.334531 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:38:55.334481    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:39:01.258641 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:01.258579    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:39:01.259056 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:01.258667    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:39:01.271754 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:01.271723    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:39:01.283479 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:01.283447    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:39:19.637927 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:19.637835    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:39:35.633381 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:35.633333    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:39:43.040337 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:43.040293    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"]
Jul 16 05:39:43.918009 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:43.917972    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"]
Jul 16 05:39:45.070122 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:45.070084    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-139702663265200"]
Jul 16 05:39:45.073687 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:45.073662    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663265200"
Jul 16 05:39:45.076400 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:45.076371    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2zcq8\""
Jul 16 05:39:45.080963 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:45.080932    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-139702663265200"]
Jul 16 05:39:45.197944 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:45.197896    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4thd9\" (UniqueName: \"kubernetes.io/projected/da416321-74c3-4269-8153-286b8348b8c6-kube-api-access-4thd9\") pod \"test-curl-7600-139702663265200\" (UID: \"da416321-74c3-4269-8153-286b8348b8c6\") " pod="opendatahub/test-curl-7600-139702663265200"
Jul 16 05:39:45.298868 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:45.298819    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4thd9\" (UniqueName: \"kubernetes.io/projected/da416321-74c3-4269-8153-286b8348b8c6-kube-api-access-4thd9\") pod \"test-curl-7600-139702663265200\" (UID: \"da416321-74c3-4269-8153-286b8348b8c6\") " pod="opendatahub/test-curl-7600-139702663265200"
Jul 16 05:39:45.306941 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:45.306904    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4thd9\" (UniqueName: \"kubernetes.io/projected/da416321-74c3-4269-8153-286b8348b8c6-kube-api-access-4thd9\") pod \"test-curl-7600-139702663265200\" (UID: \"da416321-74c3-4269-8153-286b8348b8c6\") " pod="opendatahub/test-curl-7600-139702663265200"
Jul 16 05:39:45.386064 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:45.385969    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663265200"
Jul 16 05:39:45.542992 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:45.542960    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-139702663265200"]
Jul 16 05:39:45.545227 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:39:45.545193    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda416321_74c3_4269_8153_286b8348b8c6.slice/crio-62dfc94619ec2b6868e0564cad70f47fe8055299d1eb3d71b23aace1fc73da3d WatchSource:0}: Error finding container 62dfc94619ec2b6868e0564cad70f47fe8055299d1eb3d71b23aace1fc73da3d: Status 404 returned error can't find the container with id 62dfc94619ec2b6868e0564cad70f47fe8055299d1eb3d71b23aace1fc73da3d
Jul 16 05:39:45.546491 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:45.546472    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 05:39:46.551237 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:46.551143    2565 generic.go:356] "Generic (PLEG): container finished" podID="da416321-74c3-4269-8153-286b8348b8c6" containerID="5dc5af3fc2a3aaa037ed214bcb95b823b577c82841a5ccf13b3412d43f6e85a9" exitCode=0
Jul 16 05:39:46.551660 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:46.551240    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-139702663265200" event={"ID":"da416321-74c3-4269-8153-286b8348b8c6","Type":"ContainerDied","Data":"5dc5af3fc2a3aaa037ed214bcb95b823b577c82841a5ccf13b3412d43f6e85a9"}
Jul 16 05:39:46.551660 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:46.551291    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-139702663265200" event={"ID":"da416321-74c3-4269-8153-286b8348b8c6","Type":"ContainerStarted","Data":"62dfc94619ec2b6868e0564cad70f47fe8055299d1eb3d71b23aace1fc73da3d"}
Jul 16 05:39:47.703330 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:47.703286    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663265200"
Jul 16 05:39:47.736104 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:47.736077    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7600-139702663265200_da416321-74c3-4269-8153-286b8348b8c6/test-curl-7600-139702663265200/0.log"
Jul 16 05:39:47.760128 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:47.760071    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-139702663265200"]
Jul 16 05:39:47.762542 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:47.762515    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-139702663265200"]
Jul 16 05:39:47.824881 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:47.824786    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4thd9\" (UniqueName: \"kubernetes.io/projected/da416321-74c3-4269-8153-286b8348b8c6-kube-api-access-4thd9\") pod \"da416321-74c3-4269-8153-286b8348b8c6\" (UID: \"da416321-74c3-4269-8153-286b8348b8c6\") "
Jul 16 05:39:47.827308 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:47.827268    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da416321-74c3-4269-8153-286b8348b8c6-kube-api-access-4thd9" (OuterVolumeSpecName: "kube-api-access-4thd9") pod "da416321-74c3-4269-8153-286b8348b8c6" (UID: "da416321-74c3-4269-8153-286b8348b8c6"). InnerVolumeSpecName "kube-api-access-4thd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:39:47.894845 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:47.894805    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-139702663263024"]
Jul 16 05:39:47.895251 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:47.895230    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da416321-74c3-4269-8153-286b8348b8c6" containerName="test-curl-7600-139702663265200"
Jul 16 05:39:47.895251 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:47.895246    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da416321-74c3-4269-8153-286b8348b8c6" containerName="test-curl-7600-139702663265200"
Jul 16 05:39:47.895372 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:47.895318    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da416321-74c3-4269-8153-286b8348b8c6" containerName="test-curl-7600-139702663265200"
Jul 16 05:39:47.898505 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:47.898483    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663263024"
Jul 16 05:39:47.904394 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:47.904361    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-139702663263024"]
Jul 16 05:39:47.926026 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:47.925987    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4thd9\" (UniqueName: \"kubernetes.io/projected/da416321-74c3-4269-8153-286b8348b8c6-kube-api-access-4thd9\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:39:48.026670 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:48.026630    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6r58\" (UniqueName: \"kubernetes.io/projected/e950d1a8-86d4-4b3a-bc3e-cb4997f0a779-kube-api-access-j6r58\") pod \"test-curl-7600-139702663263024\" (UID: \"e950d1a8-86d4-4b3a-bc3e-cb4997f0a779\") " pod="opendatahub/test-curl-7600-139702663263024"
Jul 16 05:39:48.127226 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:48.127117    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6r58\" (UniqueName: \"kubernetes.io/projected/e950d1a8-86d4-4b3a-bc3e-cb4997f0a779-kube-api-access-j6r58\") pod \"test-curl-7600-139702663263024\" (UID: \"e950d1a8-86d4-4b3a-bc3e-cb4997f0a779\") " pod="opendatahub/test-curl-7600-139702663263024"
Jul 16 05:39:48.134875 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:48.134831    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6r58\" (UniqueName: \"kubernetes.io/projected/e950d1a8-86d4-4b3a-bc3e-cb4997f0a779-kube-api-access-j6r58\") pod \"test-curl-7600-139702663263024\" (UID: \"e950d1a8-86d4-4b3a-bc3e-cb4997f0a779\") " pod="opendatahub/test-curl-7600-139702663263024"
Jul 16 05:39:48.209554 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:48.209508    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663263024"
Jul 16 05:39:48.359786 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:48.359746    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-139702663263024"]
Jul 16 05:39:48.362257 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:39:48.362228    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode950d1a8_86d4_4b3a_bc3e_cb4997f0a779.slice/crio-3ce50cf9bf4a6a5ae8a349dd4881d205624e8c318aba10de1297585169cd71aa WatchSource:0}: Error finding container 3ce50cf9bf4a6a5ae8a349dd4881d205624e8c318aba10de1297585169cd71aa: Status 404 returned error can't find the container with id 3ce50cf9bf4a6a5ae8a349dd4881d205624e8c318aba10de1297585169cd71aa
Jul 16 05:39:48.563975 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:48.563929    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-139702663263024" event={"ID":"e950d1a8-86d4-4b3a-bc3e-cb4997f0a779","Type":"ContainerStarted","Data":"3ce50cf9bf4a6a5ae8a349dd4881d205624e8c318aba10de1297585169cd71aa"}
Jul 16 05:39:48.565313 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:48.565284    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663265200"
Jul 16 05:39:48.565476 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:48.565331    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62dfc94619ec2b6868e0564cad70f47fe8055299d1eb3d71b23aace1fc73da3d"
Jul 16 05:39:48.575891 ip-10-0-138-199 kubenswrapper[2565]: E0716 05:39:48.575854    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7600-139702663265200\" is forbidden: User \"system:node:ip-10-0-138-199.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-199.ec2.internal' and this object" podUID="da416321-74c3-4269-8153-286b8348b8c6" pod="opendatahub/test-curl-7600-139702663265200"
Jul 16 05:39:49.142444 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:49.142398    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da416321-74c3-4269-8153-286b8348b8c6" path="/var/lib/kubelet/pods/da416321-74c3-4269-8153-286b8348b8c6/volumes"
Jul 16 05:39:49.438699 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:49.438563    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:39:49.572971 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:49.572932    2565 generic.go:356] "Generic (PLEG): container finished" podID="e950d1a8-86d4-4b3a-bc3e-cb4997f0a779" containerID="4ddc61d9f2e02a0cbb03da5f165725c1db81deb571e70cbc15adcdaa18af416f" exitCode=0
Jul 16 05:39:49.573163 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:49.573032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-139702663263024" event={"ID":"e950d1a8-86d4-4b3a-bc3e-cb4997f0a779","Type":"ContainerDied","Data":"4ddc61d9f2e02a0cbb03da5f165725c1db81deb571e70cbc15adcdaa18af416f"}
Jul 16 05:39:50.722965 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:50.722930    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663263024"
Jul 16 05:39:50.741763 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:50.741730    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7600-139702663263024_e950d1a8-86d4-4b3a-bc3e-cb4997f0a779/test-curl-7600-139702663263024/0.log"
Jul 16 05:39:50.766621 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:50.766540    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-139702663263024"]
Jul 16 05:39:50.770539 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:50.770500    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-139702663263024"]
Jul 16 05:39:50.854799 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:50.854699    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6r58\" (UniqueName: \"kubernetes.io/projected/e950d1a8-86d4-4b3a-bc3e-cb4997f0a779-kube-api-access-j6r58\") pod \"e950d1a8-86d4-4b3a-bc3e-cb4997f0a779\" (UID: \"e950d1a8-86d4-4b3a-bc3e-cb4997f0a779\") "
Jul 16 05:39:50.857088 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:50.857060    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e950d1a8-86d4-4b3a-bc3e-cb4997f0a779-kube-api-access-j6r58" (OuterVolumeSpecName: "kube-api-access-j6r58") pod "e950d1a8-86d4-4b3a-bc3e-cb4997f0a779" (UID: "e950d1a8-86d4-4b3a-bc3e-cb4997f0a779"). InnerVolumeSpecName "kube-api-access-j6r58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:39:50.936559 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:50.936522    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-139702663000400"]
Jul 16 05:39:50.936967 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:50.936951    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e950d1a8-86d4-4b3a-bc3e-cb4997f0a779" containerName="test-curl-7600-139702663263024"
Jul 16 05:39:50.937010 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:50.936968    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e950d1a8-86d4-4b3a-bc3e-cb4997f0a779" containerName="test-curl-7600-139702663263024"
Jul 16 05:39:50.937062 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:50.937050    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e950d1a8-86d4-4b3a-bc3e-cb4997f0a779" containerName="test-curl-7600-139702663263024"
Jul 16 05:39:50.940290 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:50.940263    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663000400"
Jul 16 05:39:50.945811 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:50.945780    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-139702663000400"]
Jul 16 05:39:50.955638 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:50.955567    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6r58\" (UniqueName: \"kubernetes.io/projected/e950d1a8-86d4-4b3a-bc3e-cb4997f0a779-kube-api-access-j6r58\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:39:51.057060 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:51.057018    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47r6c\" (UniqueName: \"kubernetes.io/projected/ebc62d1a-3e28-48a5-85b6-6aaba8b46574-kube-api-access-47r6c\") pod \"test-curl-7600-139702663000400\" (UID: \"ebc62d1a-3e28-48a5-85b6-6aaba8b46574\") " pod="opendatahub/test-curl-7600-139702663000400"
Jul 16 05:39:51.143046 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:51.142955    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e950d1a8-86d4-4b3a-bc3e-cb4997f0a779" path="/var/lib/kubelet/pods/e950d1a8-86d4-4b3a-bc3e-cb4997f0a779/volumes"
Jul 16 05:39:51.158462 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:51.158426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47r6c\" (UniqueName: \"kubernetes.io/projected/ebc62d1a-3e28-48a5-85b6-6aaba8b46574-kube-api-access-47r6c\") pod \"test-curl-7600-139702663000400\" (UID: \"ebc62d1a-3e28-48a5-85b6-6aaba8b46574\") " pod="opendatahub/test-curl-7600-139702663000400"
Jul 16 05:39:51.166434 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:51.166400    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47r6c\" (UniqueName: \"kubernetes.io/projected/ebc62d1a-3e28-48a5-85b6-6aaba8b46574-kube-api-access-47r6c\") pod \"test-curl-7600-139702663000400\" (UID: \"ebc62d1a-3e28-48a5-85b6-6aaba8b46574\") " pod="opendatahub/test-curl-7600-139702663000400"
Jul 16 05:39:51.252475 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:51.252422    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663000400"
Jul 16 05:39:51.402751 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:51.402712    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-139702663000400"]
Jul 16 05:39:51.404902 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:39:51.404872    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebc62d1a_3e28_48a5_85b6_6aaba8b46574.slice/crio-b0dbb00b57443560dc6d8ec8cf75a0c252d9a96f90e28e9bc354e7b3f4e1ee69 WatchSource:0}: Error finding container b0dbb00b57443560dc6d8ec8cf75a0c252d9a96f90e28e9bc354e7b3f4e1ee69: Status 404 returned error can't find the container with id b0dbb00b57443560dc6d8ec8cf75a0c252d9a96f90e28e9bc354e7b3f4e1ee69
Jul 16 05:39:51.585969 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:51.585938    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663263024"
Jul 16 05:39:51.586182 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:51.586000    2565 scope.go:117] "RemoveContainer" containerID="4ddc61d9f2e02a0cbb03da5f165725c1db81deb571e70cbc15adcdaa18af416f"
Jul 16 05:39:51.587544 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:51.587393    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-139702663000400" event={"ID":"ebc62d1a-3e28-48a5-85b6-6aaba8b46574","Type":"ContainerStarted","Data":"b0dbb00b57443560dc6d8ec8cf75a0c252d9a96f90e28e9bc354e7b3f4e1ee69"}
Jul 16 05:39:52.596285 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:52.596243    2565 generic.go:356] "Generic (PLEG): container finished" podID="ebc62d1a-3e28-48a5-85b6-6aaba8b46574" containerID="c4253bb9a28e466f0387b1cd90b8769156e3468134dcad7c3145beda179edc91" exitCode=0
Jul 16 05:39:52.596772 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:52.596298    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-139702663000400" event={"ID":"ebc62d1a-3e28-48a5-85b6-6aaba8b46574","Type":"ContainerDied","Data":"c4253bb9a28e466f0387b1cd90b8769156e3468134dcad7c3145beda179edc91"}
Jul 16 05:39:53.750062 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:53.750029    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663000400"
Jul 16 05:39:53.769159 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:53.769130    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7600-139702663000400_ebc62d1a-3e28-48a5-85b6-6aaba8b46574/test-curl-7600-139702663000400/0.log"
Jul 16 05:39:53.795324 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:53.795287    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-139702663000400"]
Jul 16 05:39:53.798902 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:53.798871    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-139702663000400"]
Jul 16 05:39:53.885756 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:53.885648    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47r6c\" (UniqueName: \"kubernetes.io/projected/ebc62d1a-3e28-48a5-85b6-6aaba8b46574-kube-api-access-47r6c\") pod \"ebc62d1a-3e28-48a5-85b6-6aaba8b46574\" (UID: \"ebc62d1a-3e28-48a5-85b6-6aaba8b46574\") "
Jul 16 05:39:53.888000 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:53.887961    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc62d1a-3e28-48a5-85b6-6aaba8b46574-kube-api-access-47r6c" (OuterVolumeSpecName: "kube-api-access-47r6c") pod "ebc62d1a-3e28-48a5-85b6-6aaba8b46574" (UID: "ebc62d1a-3e28-48a5-85b6-6aaba8b46574"). InnerVolumeSpecName "kube-api-access-47r6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:39:53.927511 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:53.927470    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-139702663000400"]
Jul 16 05:39:53.927993 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:53.927971    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc62d1a-3e28-48a5-85b6-6aaba8b46574" containerName="test-curl-7600-139702663000400"
Jul 16 05:39:53.928095 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:53.927996    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc62d1a-3e28-48a5-85b6-6aaba8b46574" containerName="test-curl-7600-139702663000400"
Jul 16 05:39:53.928180 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:53.928167    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc62d1a-3e28-48a5-85b6-6aaba8b46574" containerName="test-curl-7600-139702663000400"
Jul 16 05:39:53.931412 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:53.931383    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663000400"
Jul 16 05:39:53.936216 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:53.936188    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-139702663000400"]
Jul 16 05:39:53.986809 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:53.986763    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx8kv\" (UniqueName: \"kubernetes.io/projected/c629c081-76bf-4906-a3f2-5f97296de41c-kube-api-access-hx8kv\") pod \"test-curl-7600-139702663000400\" (UID: \"c629c081-76bf-4906-a3f2-5f97296de41c\") " pod="opendatahub/test-curl-7600-139702663000400"
Jul 16 05:39:53.987008 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:53.986852    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47r6c\" (UniqueName: \"kubernetes.io/projected/ebc62d1a-3e28-48a5-85b6-6aaba8b46574-kube-api-access-47r6c\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:39:54.087677 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:54.087631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx8kv\" (UniqueName: \"kubernetes.io/projected/c629c081-76bf-4906-a3f2-5f97296de41c-kube-api-access-hx8kv\") pod \"test-curl-7600-139702663000400\" (UID: \"c629c081-76bf-4906-a3f2-5f97296de41c\") " pod="opendatahub/test-curl-7600-139702663000400"
Jul 16 05:39:54.096078 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:54.096052    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx8kv\" (UniqueName: \"kubernetes.io/projected/c629c081-76bf-4906-a3f2-5f97296de41c-kube-api-access-hx8kv\") pod \"test-curl-7600-139702663000400\" (UID: \"c629c081-76bf-4906-a3f2-5f97296de41c\") " pod="opendatahub/test-curl-7600-139702663000400"
Jul 16 05:39:54.242937 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:54.242896    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663000400"
Jul 16 05:39:54.392820 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:54.392783    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-139702663000400"]
Jul 16 05:39:54.394173 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:39:54.394144    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc629c081_76bf_4906_a3f2_5f97296de41c.slice/crio-9be3ed08e9d09141f1ec3cbaf683a7a914e7f929f31b4dd8f21c8b07908bc210 WatchSource:0}: Error finding container 9be3ed08e9d09141f1ec3cbaf683a7a914e7f929f31b4dd8f21c8b07908bc210: Status 404 returned error can't find the container with id 9be3ed08e9d09141f1ec3cbaf683a7a914e7f929f31b4dd8f21c8b07908bc210
Jul 16 05:39:54.610940 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:54.610832    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-139702663000400" event={"ID":"c629c081-76bf-4906-a3f2-5f97296de41c","Type":"ContainerStarted","Data":"9be3ed08e9d09141f1ec3cbaf683a7a914e7f929f31b4dd8f21c8b07908bc210"}
Jul 16 05:39:54.612453 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:54.612421    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663000400"
Jul 16 05:39:54.612631 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:54.612478    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0dbb00b57443560dc6d8ec8cf75a0c252d9a96f90e28e9bc354e7b3f4e1ee69"
Jul 16 05:39:54.627490 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:54.627445    2565 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7600-139702663000400" oldPodUID="ebc62d1a-3e28-48a5-85b6-6aaba8b46574" podUID="c629c081-76bf-4906-a3f2-5f97296de41c"
Jul 16 05:39:55.142802 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:55.142761    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebc62d1a-3e28-48a5-85b6-6aaba8b46574" path="/var/lib/kubelet/pods/ebc62d1a-3e28-48a5-85b6-6aaba8b46574/volumes"
Jul 16 05:39:55.620703 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:55.620670    2565 generic.go:356] "Generic (PLEG): container finished" podID="c629c081-76bf-4906-a3f2-5f97296de41c" containerID="882581937aac9a9563836e27c327e0a0a501d0c025bedeb58a3ac209b2232042" exitCode=0
Jul 16 05:39:55.620905 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:55.620727    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-139702663000400" event={"ID":"c629c081-76bf-4906-a3f2-5f97296de41c","Type":"ContainerDied","Data":"882581937aac9a9563836e27c327e0a0a501d0c025bedeb58a3ac209b2232042"}
Jul 16 05:39:56.772065 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:56.772034    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663000400"
Jul 16 05:39:56.793187 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:56.793151    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7600-139702663000400_c629c081-76bf-4906-a3f2-5f97296de41c/test-curl-7600-139702663000400/0.log"
Jul 16 05:39:56.817270 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:56.817232    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-139702663000400"]
Jul 16 05:39:56.824149 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:56.824103    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-139702663000400"]
Jul 16 05:39:56.915753 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:56.915652    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx8kv\" (UniqueName: \"kubernetes.io/projected/c629c081-76bf-4906-a3f2-5f97296de41c-kube-api-access-hx8kv\") pod \"c629c081-76bf-4906-a3f2-5f97296de41c\" (UID: \"c629c081-76bf-4906-a3f2-5f97296de41c\") "
Jul 16 05:39:56.918024 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:56.917987    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c629c081-76bf-4906-a3f2-5f97296de41c-kube-api-access-hx8kv" (OuterVolumeSpecName: "kube-api-access-hx8kv") pod "c629c081-76bf-4906-a3f2-5f97296de41c" (UID: "c629c081-76bf-4906-a3f2-5f97296de41c"). InnerVolumeSpecName "kube-api-access-hx8kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:39:56.948970 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:56.948930    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-139702663001408"]
Jul 16 05:39:56.949357 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:56.949343    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c629c081-76bf-4906-a3f2-5f97296de41c" containerName="test-curl-7600-139702663000400"
Jul 16 05:39:56.949419 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:56.949359    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c629c081-76bf-4906-a3f2-5f97296de41c" containerName="test-curl-7600-139702663000400"
Jul 16 05:39:56.949460 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:56.949426    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c629c081-76bf-4906-a3f2-5f97296de41c" containerName="test-curl-7600-139702663000400"
Jul 16 05:39:56.952501 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:56.952477    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663001408"
Jul 16 05:39:56.958464 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:56.958433    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-139702663001408"]
Jul 16 05:39:57.016900 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:57.016849    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-892wx\" (UniqueName: \"kubernetes.io/projected/cbf7401d-58e7-4252-b098-e04442219036-kube-api-access-892wx\") pod \"test-curl-7600-139702663001408\" (UID: \"cbf7401d-58e7-4252-b098-e04442219036\") " pod="opendatahub/test-curl-7600-139702663001408"
Jul 16 05:39:57.017128 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:57.017052    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx8kv\" (UniqueName: \"kubernetes.io/projected/c629c081-76bf-4906-a3f2-5f97296de41c-kube-api-access-hx8kv\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:39:57.117840 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:57.117796    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-892wx\" (UniqueName: \"kubernetes.io/projected/cbf7401d-58e7-4252-b098-e04442219036-kube-api-access-892wx\") pod \"test-curl-7600-139702663001408\" (UID: \"cbf7401d-58e7-4252-b098-e04442219036\") " pod="opendatahub/test-curl-7600-139702663001408"
Jul 16 05:39:57.128748 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:57.128711    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-892wx\" (UniqueName: \"kubernetes.io/projected/cbf7401d-58e7-4252-b098-e04442219036-kube-api-access-892wx\") pod \"test-curl-7600-139702663001408\" (UID: \"cbf7401d-58e7-4252-b098-e04442219036\") " pod="opendatahub/test-curl-7600-139702663001408"
Jul 16 05:39:57.142960 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:57.142927    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c629c081-76bf-4906-a3f2-5f97296de41c" path="/var/lib/kubelet/pods/c629c081-76bf-4906-a3f2-5f97296de41c/volumes"
Jul 16 05:39:57.263211 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:57.263176    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663001408"
Jul 16 05:39:57.418524 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:57.418453    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-139702663001408"]
Jul 16 05:39:57.420706 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:39:57.420664    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbf7401d_58e7_4252_b098_e04442219036.slice/crio-86986c403295c77a2044ea633a800e84f7b0293579514b1279ab0fc55848b47d WatchSource:0}: Error finding container 86986c403295c77a2044ea633a800e84f7b0293579514b1279ab0fc55848b47d: Status 404 returned error can't find the container with id 86986c403295c77a2044ea633a800e84f7b0293579514b1279ab0fc55848b47d
Jul 16 05:39:57.636139 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:57.635940    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663000400"
Jul 16 05:39:57.636139 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:57.635962    2565 scope.go:117] "RemoveContainer" containerID="882581937aac9a9563836e27c327e0a0a501d0c025bedeb58a3ac209b2232042"
Jul 16 05:39:57.659664 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:57.637339    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-139702663001408" event={"ID":"cbf7401d-58e7-4252-b098-e04442219036","Type":"ContainerStarted","Data":"86986c403295c77a2044ea633a800e84f7b0293579514b1279ab0fc55848b47d"}
Jul 16 05:39:57.659664 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:57.654403    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:39:57.659664 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:57.659113    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:39:57.697799 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:57.697772    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:39:57.702721 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:57.702694    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsc5_72b01c84-01a4-4022-b66f-95876da5da22/ovn-acl-logging/0.log"
Jul 16 05:39:57.824628 ip-10-0-138-199 kubenswrapper[2565]: E0716 05:39:57.824574    2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbf7401d_58e7_4252_b098_e04442219036.slice/crio-conmon-e2b87e2b877e4f89bb0b8f0051652be2bebb13604ff4f3761426158d84a19c9c.scope\": RecentStats: unable to find data in memory cache]"
Jul 16 05:39:58.062702 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.062657    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z" podUID="256d1ed7-4b8d-4791-a33f-b24df47937a3" containerName="main" containerID="cri-o://39d58e4952d31282e9960be5d3c882a552f59711095cff00a83da4983abb9915" gracePeriod=45
Jul 16 05:39:58.332972 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.332939    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:39:58.434406 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.434370    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-dshm\") pod \"256d1ed7-4b8d-4791-a33f-b24df47937a3\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") "
Jul 16 05:39:58.434406 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.434409    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/256d1ed7-4b8d-4791-a33f-b24df47937a3-tls-certs\") pod \"256d1ed7-4b8d-4791-a33f-b24df47937a3\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") "
Jul 16 05:39:58.434685 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.434448    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zspqb\" (UniqueName: \"kubernetes.io/projected/256d1ed7-4b8d-4791-a33f-b24df47937a3-kube-api-access-zspqb\") pod \"256d1ed7-4b8d-4791-a33f-b24df47937a3\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") "
Jul 16 05:39:58.434685 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.434475    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-model-cache\") pod \"256d1ed7-4b8d-4791-a33f-b24df47937a3\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") "
Jul 16 05:39:58.434685 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.434534    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-home\") pod \"256d1ed7-4b8d-4791-a33f-b24df47937a3\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") "
Jul 16 05:39:58.434880 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.434783    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-tmp-dir\") pod \"256d1ed7-4b8d-4791-a33f-b24df47937a3\" (UID: \"256d1ed7-4b8d-4791-a33f-b24df47937a3\") "
Jul 16 05:39:58.435027 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.434986    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "256d1ed7-4b8d-4791-a33f-b24df47937a3" (UID: "256d1ed7-4b8d-4791-a33f-b24df47937a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:39:58.435027 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.435017    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-home" (OuterVolumeSpecName: "home") pod "256d1ed7-4b8d-4791-a33f-b24df47937a3" (UID: "256d1ed7-4b8d-4791-a33f-b24df47937a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:39:58.435668 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.435325    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-tmp-dir\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:39:58.435668 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.435354    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-home\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:39:58.435668 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.435587    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "256d1ed7-4b8d-4791-a33f-b24df47937a3" (UID: "256d1ed7-4b8d-4791-a33f-b24df47937a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:39:58.437118 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.437088    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-dshm" (OuterVolumeSpecName: "dshm") pod "256d1ed7-4b8d-4791-a33f-b24df47937a3" (UID: "256d1ed7-4b8d-4791-a33f-b24df47937a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:39:58.437425 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.437401    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/256d1ed7-4b8d-4791-a33f-b24df47937a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "256d1ed7-4b8d-4791-a33f-b24df47937a3" (UID: "256d1ed7-4b8d-4791-a33f-b24df47937a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 05:39:58.437512 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.437440    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256d1ed7-4b8d-4791-a33f-b24df47937a3-kube-api-access-zspqb" (OuterVolumeSpecName: "kube-api-access-zspqb") pod "256d1ed7-4b8d-4791-a33f-b24df47937a3" (UID: "256d1ed7-4b8d-4791-a33f-b24df47937a3"). InnerVolumeSpecName "kube-api-access-zspqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:39:58.536819 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.536767    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-dshm\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:39:58.536819 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.536807    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/256d1ed7-4b8d-4791-a33f-b24df47937a3-tls-certs\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:39:58.536819 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.536819    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zspqb\" (UniqueName: \"kubernetes.io/projected/256d1ed7-4b8d-4791-a33f-b24df47937a3-kube-api-access-zspqb\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:39:58.536819 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.536828    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/256d1ed7-4b8d-4791-a33f-b24df47937a3-model-cache\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:39:58.645576 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.645473    2565 generic.go:356] "Generic (PLEG): container finished" podID="256d1ed7-4b8d-4791-a33f-b24df47937a3" containerID="39d58e4952d31282e9960be5d3c882a552f59711095cff00a83da4983abb9915" exitCode=0
Jul 16 05:39:58.645576 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.645522    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"
Jul 16 05:39:58.645822 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.645568    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z" event={"ID":"256d1ed7-4b8d-4791-a33f-b24df47937a3","Type":"ContainerDied","Data":"39d58e4952d31282e9960be5d3c882a552f59711095cff00a83da4983abb9915"}
Jul 16 05:39:58.645822 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.645641    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z" event={"ID":"256d1ed7-4b8d-4791-a33f-b24df47937a3","Type":"ContainerDied","Data":"150472ea1cc91e8017440232b7e8555bf944f025b31ab456bbd8560f0fee8cb0"}
Jul 16 05:39:58.645822 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.645660    2565 scope.go:117] "RemoveContainer" containerID="39d58e4952d31282e9960be5d3c882a552f59711095cff00a83da4983abb9915"
Jul 16 05:39:58.648779 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.648748    2565 generic.go:356] "Generic (PLEG): container finished" podID="cbf7401d-58e7-4252-b098-e04442219036" containerID="e2b87e2b877e4f89bb0b8f0051652be2bebb13604ff4f3761426158d84a19c9c" exitCode=0
Jul 16 05:39:58.648962 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.648839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-139702663001408" event={"ID":"cbf7401d-58e7-4252-b098-e04442219036","Type":"ContainerDied","Data":"e2b87e2b877e4f89bb0b8f0051652be2bebb13604ff4f3761426158d84a19c9c"}
Jul 16 05:39:58.657053 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.657024    2565 scope.go:117] "RemoveContainer" containerID="39d58e4952d31282e9960be5d3c882a552f59711095cff00a83da4983abb9915"
Jul 16 05:39:58.657423 ip-10-0-138-199 kubenswrapper[2565]: E0716 05:39:58.657403    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d58e4952d31282e9960be5d3c882a552f59711095cff00a83da4983abb9915\": container with ID starting with 39d58e4952d31282e9960be5d3c882a552f59711095cff00a83da4983abb9915 not found: ID does not exist" containerID="39d58e4952d31282e9960be5d3c882a552f59711095cff00a83da4983abb9915"
Jul 16 05:39:58.657481 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.657434    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d58e4952d31282e9960be5d3c882a552f59711095cff00a83da4983abb9915"} err="failed to get container status \"39d58e4952d31282e9960be5d3c882a552f59711095cff00a83da4983abb9915\": rpc error: code = NotFound desc = could not find container \"39d58e4952d31282e9960be5d3c882a552f59711095cff00a83da4983abb9915\": container with ID starting with 39d58e4952d31282e9960be5d3c882a552f59711095cff00a83da4983abb9915 not found: ID does not exist"
Jul 16 05:39:58.675022 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.674947    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"]
Jul 16 05:39:58.678729 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.678689    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-615edb/rate-test-model-615edb-kserve-756587567f-2kq8z"]
Jul 16 05:39:58.940528 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:58.940428    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5" podUID="ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6" containerName="main" containerID="cri-o://0e8bc9a4d88c5c94e5096189c02def51d06829276128520520f0f73d81124f2e" gracePeriod=45
Jul 16 05:39:59.144223 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.144187    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="256d1ed7-4b8d-4791-a33f-b24df47937a3" path="/var/lib/kubelet/pods/256d1ed7-4b8d-4791-a33f-b24df47937a3/volumes"
Jul 16 05:39:59.207194 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.207105    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:39:59.344764 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.344722    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-model-cache\") pod \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") "
Jul 16 05:39:59.344972 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.344780    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-tls-certs\") pod \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") "
Jul 16 05:39:59.344972 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.344870    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-dshm\") pod \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") "
Jul 16 05:39:59.344972 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.344899    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-tmp-dir\") pod \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") "
Jul 16 05:39:59.344972 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.344939    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-home\") pod \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") "
Jul 16 05:39:59.345203 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.344986    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btcwl\" (UniqueName: \"kubernetes.io/projected/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-kube-api-access-btcwl\") pod \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\" (UID: \"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6\") "
Jul 16 05:39:59.345203 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.345073    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6" (UID: "ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:39:59.345311 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.345230    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-home" (OuterVolumeSpecName: "home") pod "ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6" (UID: "ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:39:59.345311 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.345285    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-model-cache\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:39:59.345428 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.345336    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6" (UID: "ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:39:59.347459 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.347427    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-dshm" (OuterVolumeSpecName: "dshm") pod "ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6" (UID: "ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 05:39:59.347640 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.347456    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-kube-api-access-btcwl" (OuterVolumeSpecName: "kube-api-access-btcwl") pod "ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6" (UID: "ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6"). InnerVolumeSpecName "kube-api-access-btcwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:39:59.347640 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.347553    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6" (UID: "ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 05:39:59.445926 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.445882    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btcwl\" (UniqueName: \"kubernetes.io/projected/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-kube-api-access-btcwl\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:39:59.445926 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.445915    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-tls-certs\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:39:59.445926 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.445926    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-dshm\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:39:59.445926 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.445933    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-tmp-dir\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:39:59.446218 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.445942    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6-home\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:39:59.657709 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.657672    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6" containerID="0e8bc9a4d88c5c94e5096189c02def51d06829276128520520f0f73d81124f2e" exitCode=0
Jul 16 05:39:59.657904 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.657717    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"
Jul 16 05:39:59.657904 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.657762    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5" event={"ID":"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6","Type":"ContainerDied","Data":"0e8bc9a4d88c5c94e5096189c02def51d06829276128520520f0f73d81124f2e"}
Jul 16 05:39:59.657904 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.657808    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5" event={"ID":"ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6","Type":"ContainerDied","Data":"d7986f5776ba78485a44c05b25c51d7867e6bd220458680b368a23cc0cc5a941"}
Jul 16 05:39:59.657904 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.657840    2565 scope.go:117] "RemoveContainer" containerID="0e8bc9a4d88c5c94e5096189c02def51d06829276128520520f0f73d81124f2e"
Jul 16 05:39:59.675314 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.675291    2565 scope.go:117] "RemoveContainer" containerID="0e8bc9a4d88c5c94e5096189c02def51d06829276128520520f0f73d81124f2e"
Jul 16 05:39:59.675722 ip-10-0-138-199 kubenswrapper[2565]: E0716 05:39:59.675699    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e8bc9a4d88c5c94e5096189c02def51d06829276128520520f0f73d81124f2e\": container with ID starting with 0e8bc9a4d88c5c94e5096189c02def51d06829276128520520f0f73d81124f2e not found: ID does not exist" containerID="0e8bc9a4d88c5c94e5096189c02def51d06829276128520520f0f73d81124f2e"
Jul 16 05:39:59.675786 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.675734    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e8bc9a4d88c5c94e5096189c02def51d06829276128520520f0f73d81124f2e"} err="failed to get container status \"0e8bc9a4d88c5c94e5096189c02def51d06829276128520520f0f73d81124f2e\": rpc error: code = NotFound desc = could not find container \"0e8bc9a4d88c5c94e5096189c02def51d06829276128520520f0f73d81124f2e\": container with ID starting with 0e8bc9a4d88c5c94e5096189c02def51d06829276128520520f0f73d81124f2e not found: ID does not exist"
Jul 16 05:39:59.682298 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.682246    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"]
Jul 16 05:39:59.686784 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.686740    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e36375/rate-test-model-e36375-kserve-6f68f444ff-7lhb5"]
Jul 16 05:39:59.822232 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.822195    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663001408"
Jul 16 05:39:59.840564 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.840528    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7600-139702663001408_cbf7401d-58e7-4252-b098-e04442219036/test-curl-7600-139702663001408/0.log"
Jul 16 05:39:59.863832 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.863792    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-139702663001408"]
Jul 16 05:39:59.867320 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.867286    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-139702663001408"]
Jul 16 05:39:59.951243 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.951141    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-892wx\" (UniqueName: \"kubernetes.io/projected/cbf7401d-58e7-4252-b098-e04442219036-kube-api-access-892wx\") pod \"cbf7401d-58e7-4252-b098-e04442219036\" (UID: \"cbf7401d-58e7-4252-b098-e04442219036\") "
Jul 16 05:39:59.953536 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.953503    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf7401d-58e7-4252-b098-e04442219036-kube-api-access-892wx" (OuterVolumeSpecName: "kube-api-access-892wx") pod "cbf7401d-58e7-4252-b098-e04442219036" (UID: "cbf7401d-58e7-4252-b098-e04442219036"). InnerVolumeSpecName "kube-api-access-892wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:39:59.994474 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.994428    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-139702663001408"]
Jul 16 05:39:59.994857 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.994842    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="256d1ed7-4b8d-4791-a33f-b24df47937a3" containerName="main"
Jul 16 05:39:59.994912 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.994858    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="256d1ed7-4b8d-4791-a33f-b24df47937a3" containerName="main"
Jul 16 05:39:59.994912 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.994870    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6" containerName="main"
Jul 16 05:39:59.994912 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.994876    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6" containerName="main"
Jul 16 05:39:59.994912 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.994899    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf7401d-58e7-4252-b098-e04442219036" containerName="test-curl-7600-139702663001408"
Jul 16 05:39:59.994912 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.994907    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf7401d-58e7-4252-b098-e04442219036" containerName="test-curl-7600-139702663001408"
Jul 16 05:39:59.995071 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.994966    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6" containerName="main"
Jul 16 05:39:59.995071 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.994975    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbf7401d-58e7-4252-b098-e04442219036" containerName="test-curl-7600-139702663001408"
Jul 16 05:39:59.995071 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.994985    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="256d1ed7-4b8d-4791-a33f-b24df47937a3" containerName="main"
Jul 16 05:39:59.998259 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:39:59.998234    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663001408"
Jul 16 05:40:00.003248 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:00.003211    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-139702663001408"]
Jul 16 05:40:00.052205 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:00.052167    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-892wx\" (UniqueName: \"kubernetes.io/projected/cbf7401d-58e7-4252-b098-e04442219036-kube-api-access-892wx\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:40:00.152774 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:00.152732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2rts\" (UniqueName: \"kubernetes.io/projected/474f4f3d-a59a-478b-b072-8f34d6e1552f-kube-api-access-l2rts\") pod \"test-curl-7600-139702663001408\" (UID: \"474f4f3d-a59a-478b-b072-8f34d6e1552f\") " pod="opendatahub/test-curl-7600-139702663001408"
Jul 16 05:40:00.254247 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:00.254209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2rts\" (UniqueName: \"kubernetes.io/projected/474f4f3d-a59a-478b-b072-8f34d6e1552f-kube-api-access-l2rts\") pod \"test-curl-7600-139702663001408\" (UID: \"474f4f3d-a59a-478b-b072-8f34d6e1552f\") " pod="opendatahub/test-curl-7600-139702663001408"
Jul 16 05:40:00.262816 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:00.262774    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2rts\" (UniqueName: \"kubernetes.io/projected/474f4f3d-a59a-478b-b072-8f34d6e1552f-kube-api-access-l2rts\") pod \"test-curl-7600-139702663001408\" (UID: \"474f4f3d-a59a-478b-b072-8f34d6e1552f\") " pod="opendatahub/test-curl-7600-139702663001408"
Jul 16 05:40:00.310464 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:00.310416    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663001408"
Jul 16 05:40:00.462240 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:00.462214    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-139702663001408"]
Jul 16 05:40:00.464159 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:40:00.464104    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod474f4f3d_a59a_478b_b072_8f34d6e1552f.slice/crio-f00e1e6a92cda5827f73195eca2a5434d65b146f62f310763187bbf57e579b13 WatchSource:0}: Error finding container f00e1e6a92cda5827f73195eca2a5434d65b146f62f310763187bbf57e579b13: Status 404 returned error can't find the container with id f00e1e6a92cda5827f73195eca2a5434d65b146f62f310763187bbf57e579b13
Jul 16 05:40:00.666184 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:00.666095    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663001408"
Jul 16 05:40:00.666184 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:00.666114    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86986c403295c77a2044ea633a800e84f7b0293579514b1279ab0fc55848b47d"
Jul 16 05:40:00.667500 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:00.667469    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-139702663001408" event={"ID":"474f4f3d-a59a-478b-b072-8f34d6e1552f","Type":"ContainerStarted","Data":"f00e1e6a92cda5827f73195eca2a5434d65b146f62f310763187bbf57e579b13"}
Jul 16 05:40:00.678129 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:00.678097    2565 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7600-139702663001408" oldPodUID="cbf7401d-58e7-4252-b098-e04442219036" podUID="474f4f3d-a59a-478b-b072-8f34d6e1552f"
Jul 16 05:40:01.142555 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:01.142512    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6" path="/var/lib/kubelet/pods/ad0f5ffc-0978-4ae4-899f-a93ddb17b9a6/volumes"
Jul 16 05:40:01.142974 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:01.142882    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbf7401d-58e7-4252-b098-e04442219036" path="/var/lib/kubelet/pods/cbf7401d-58e7-4252-b098-e04442219036/volumes"
Jul 16 05:40:01.675059 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:01.675022    2565 generic.go:356] "Generic (PLEG): container finished" podID="474f4f3d-a59a-478b-b072-8f34d6e1552f" containerID="d74192bba8767743ebf1640861acb653397642578b138f734d7bad305bdd1643" exitCode=0
Jul 16 05:40:01.675261 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:01.675111    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-139702663001408" event={"ID":"474f4f3d-a59a-478b-b072-8f34d6e1552f","Type":"ContainerDied","Data":"d74192bba8767743ebf1640861acb653397642578b138f734d7bad305bdd1643"}
Jul 16 05:40:02.839885 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:02.839847    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663001408"
Jul 16 05:40:02.859174 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:02.859135    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7600-139702663001408_474f4f3d-a59a-478b-b072-8f34d6e1552f/test-curl-7600-139702663001408/0.log"
Jul 16 05:40:02.881773 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:02.881728    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-139702663001408"]
Jul 16 05:40:02.887125 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:02.887072    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-139702663001408"]
Jul 16 05:40:02.979154 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:02.979054    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2rts\" (UniqueName: \"kubernetes.io/projected/474f4f3d-a59a-478b-b072-8f34d6e1552f-kube-api-access-l2rts\") pod \"474f4f3d-a59a-478b-b072-8f34d6e1552f\" (UID: \"474f4f3d-a59a-478b-b072-8f34d6e1552f\") "
Jul 16 05:40:02.981496 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:02.981449    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474f4f3d-a59a-478b-b072-8f34d6e1552f-kube-api-access-l2rts" (OuterVolumeSpecName: "kube-api-access-l2rts") pod "474f4f3d-a59a-478b-b072-8f34d6e1552f" (UID: "474f4f3d-a59a-478b-b072-8f34d6e1552f"). InnerVolumeSpecName "kube-api-access-l2rts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 05:40:03.080121 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:03.080072    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2rts\" (UniqueName: \"kubernetes.io/projected/474f4f3d-a59a-478b-b072-8f34d6e1552f-kube-api-access-l2rts\") on node \"ip-10-0-138-199.ec2.internal\" DevicePath \"\""
Jul 16 05:40:03.142029 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:03.141987    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="474f4f3d-a59a-478b-b072-8f34d6e1552f" path="/var/lib/kubelet/pods/474f4f3d-a59a-478b-b072-8f34d6e1552f/volumes"
Jul 16 05:40:03.688795 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:03.688760    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-139702663001408"
Jul 16 05:40:03.688975 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:03.688807    2565 scope.go:117] "RemoveContainer" containerID="d74192bba8767743ebf1640861acb653397642578b138f734d7bad305bdd1643"
Jul 16 05:40:16.828507 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:16.828466    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:40:43.326138 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:43.326087    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-78gs7"]
Jul 16 05:40:55.699560 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:40:55.699473    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7fc69f979d-ltt5g_c6c86211-b48d-46da-8ad0-3a0a28759bb9/authorino/0.log"
Jul 16 05:41:02.319833 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:02.319799    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dddb677c8-xhhkb_df4937bc-c6ba-494d-83a0-b261dc5ef290/copy-manifests/0.log"
Jul 16 05:41:02.326421 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:02.326385    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dddb677c8-xhhkb_df4937bc-c6ba-494d-83a0-b261dc5ef290/manager/0.log"
Jul 16 05:41:05.378305 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:05.378268    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw_b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0/util/0.log"
Jul 16 05:41:05.385490 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:05.385461    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw_b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0/pull/0.log"
Jul 16 05:41:05.392544 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:05.392518    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw_b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0/extract/0.log"
Jul 16 05:41:05.520235 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:05.520199    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv_82315cbd-b165-4552-8f50-301a9738f375/util/0.log"
Jul 16 05:41:05.527109 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:05.527086    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv_82315cbd-b165-4552-8f50-301a9738f375/pull/0.log"
Jul 16 05:41:05.533424 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:05.533397    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv_82315cbd-b165-4552-8f50-301a9738f375/extract/0.log"
Jul 16 05:41:05.651021 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:05.650931    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7_67b252d5-f05a-405e-8f7c-42c087adc22d/extract/0.log"
Jul 16 05:41:05.657300 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:05.657278    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7_67b252d5-f05a-405e-8f7c-42c087adc22d/util/0.log"
Jul 16 05:41:05.663816 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:05.663792    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7_67b252d5-f05a-405e-8f7c-42c087adc22d/pull/0.log"
Jul 16 05:41:05.785307 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:05.785280    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g_560c797e-c2b3-41f5-917a-9dcfc341e9cb/util/0.log"
Jul 16 05:41:05.792619 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:05.792565    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g_560c797e-c2b3-41f5-917a-9dcfc341e9cb/pull/0.log"
Jul 16 05:41:05.799729 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:05.799703    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g_560c797e-c2b3-41f5-917a-9dcfc341e9cb/extract/0.log"
Jul 16 05:41:05.921937 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:05.921850    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7fc69f979d-ltt5g_c6c86211-b48d-46da-8ad0-3a0a28759bb9/authorino/0.log"
Jul 16 05:41:06.678464 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:06.678432    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-78gs7_65fca02e-3e0b-4e0d-a5a3-60ca94b6f1cb/limitador/0.log"
Jul 16 05:41:07.548189 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:07.548135    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c7fc4f54f-mpm4d_a972592d-555b-436e-bee8-2ae1a60f45e0/kube-auth-proxy/0.log"
Jul 16 05:41:07.940412 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:07.940324    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-pn9n8_f3325934-58e9-43c1-a9c3-529eca6de6fd/payload-processing/0.log"
Jul 16 05:41:08.740636 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:08.740581    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4bggg_a9fcefb5-0081-4003-adf5-75fec2d7904d/main/0.log"
Jul 16 05:41:09.109285 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:09.109193    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-cbqtp_88de6eba-4106-4646-8272-aafe80fff8a0/main/0.log"
Jul 16 05:41:15.612277 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:15.612236    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4w7ht/must-gather-kb4k4"]
Jul 16 05:41:15.612779 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:15.612654    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="474f4f3d-a59a-478b-b072-8f34d6e1552f" containerName="test-curl-7600-139702663001408"
Jul 16 05:41:15.612779 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:15.612668    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="474f4f3d-a59a-478b-b072-8f34d6e1552f" containerName="test-curl-7600-139702663001408"
Jul 16 05:41:15.612779 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:15.612749    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="474f4f3d-a59a-478b-b072-8f34d6e1552f" containerName="test-curl-7600-139702663001408"
Jul 16 05:41:15.616076 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:15.616053    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4w7ht/must-gather-kb4k4"
Jul 16 05:41:15.618531 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:15.618508    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4w7ht\"/\"kube-root-ca.crt\""
Jul 16 05:41:15.618704 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:15.618685    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4w7ht\"/\"openshift-service-ca.crt\""
Jul 16 05:41:15.619524 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:15.619509    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4w7ht\"/\"default-dockercfg-wmljj\""
Jul 16 05:41:15.624749 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:15.624714    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4w7ht/must-gather-kb4k4"]
Jul 16 05:41:15.658878 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:15.658835    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a293a0dc-d81d-4f3e-a6ef-181ccfd1288b-must-gather-output\") pod \"must-gather-kb4k4\" (UID: \"a293a0dc-d81d-4f3e-a6ef-181ccfd1288b\") " pod="openshift-must-gather-4w7ht/must-gather-kb4k4"
Jul 16 05:41:15.659077 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:15.658885    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h9c6\" (UniqueName: \"kubernetes.io/projected/a293a0dc-d81d-4f3e-a6ef-181ccfd1288b-kube-api-access-7h9c6\") pod \"must-gather-kb4k4\" (UID: \"a293a0dc-d81d-4f3e-a6ef-181ccfd1288b\") " pod="openshift-must-gather-4w7ht/must-gather-kb4k4"
Jul 16 05:41:15.759554 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:15.759507    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a293a0dc-d81d-4f3e-a6ef-181ccfd1288b-must-gather-output\") pod \"must-gather-kb4k4\" (UID: \"a293a0dc-d81d-4f3e-a6ef-181ccfd1288b\") " pod="openshift-must-gather-4w7ht/must-gather-kb4k4"
Jul 16 05:41:15.759763 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:15.759566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h9c6\" (UniqueName: \"kubernetes.io/projected/a293a0dc-d81d-4f3e-a6ef-181ccfd1288b-kube-api-access-7h9c6\") pod \"must-gather-kb4k4\" (UID: \"a293a0dc-d81d-4f3e-a6ef-181ccfd1288b\") " pod="openshift-must-gather-4w7ht/must-gather-kb4k4"
Jul 16 05:41:15.759982 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:15.759957    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a293a0dc-d81d-4f3e-a6ef-181ccfd1288b-must-gather-output\") pod \"must-gather-kb4k4\" (UID: \"a293a0dc-d81d-4f3e-a6ef-181ccfd1288b\") " pod="openshift-must-gather-4w7ht/must-gather-kb4k4"
Jul 16 05:41:15.767615 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:15.767553    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h9c6\" (UniqueName: \"kubernetes.io/projected/a293a0dc-d81d-4f3e-a6ef-181ccfd1288b-kube-api-access-7h9c6\") pod \"must-gather-kb4k4\" (UID: \"a293a0dc-d81d-4f3e-a6ef-181ccfd1288b\") " pod="openshift-must-gather-4w7ht/must-gather-kb4k4"
Jul 16 05:41:15.928479 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:15.928359    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4w7ht/must-gather-kb4k4"
Jul 16 05:41:16.083452 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:16.083417    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4w7ht/must-gather-kb4k4"]
Jul 16 05:41:16.085640 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:41:16.085581    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda293a0dc_d81d_4f3e_a6ef_181ccfd1288b.slice/crio-75beeee58d19068939c1aecb06746be76920fb2a629b6eea559c157e4e538299 WatchSource:0}: Error finding container 75beeee58d19068939c1aecb06746be76920fb2a629b6eea559c157e4e538299: Status 404 returned error can't find the container with id 75beeee58d19068939c1aecb06746be76920fb2a629b6eea559c157e4e538299
Jul 16 05:41:16.125482 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:16.125447    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4w7ht/must-gather-kb4k4" event={"ID":"a293a0dc-d81d-4f3e-a6ef-181ccfd1288b","Type":"ContainerStarted","Data":"75beeee58d19068939c1aecb06746be76920fb2a629b6eea559c157e4e538299"}
Jul 16 05:41:17.144269 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:17.144222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4w7ht/must-gather-kb4k4" event={"ID":"a293a0dc-d81d-4f3e-a6ef-181ccfd1288b","Type":"ContainerStarted","Data":"84e8105ec1acef42f78313be1929ac04bfc4937684b1ec15ef29cdad15c2fc88"}
Jul 16 05:41:18.148663 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:18.148619    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4w7ht/must-gather-kb4k4" event={"ID":"a293a0dc-d81d-4f3e-a6ef-181ccfd1288b","Type":"ContainerStarted","Data":"5d9b463e8a96f9066a8e8757d1748ee835b0ea0ff9fa4435ef16a69e989a06a7"}
Jul 16 05:41:18.165030 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:18.164943    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4w7ht/must-gather-kb4k4" podStartSLOduration=2.30378897 podStartE2EDuration="3.164919357s" totalImagesPullingTime="861.130387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:41:18.164374562 +0000 UTC m=+3081.580329637" watchObservedRunningTime="2026-07-16 05:41:18.164919357 +0000 UTC m=+3081.580874422"
Jul 16 05:41:18.584498 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:18.584460    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dmm65_74612e83-5ffe-4c5d-830b-8e3f83fd8579/global-pull-secret-syncer/0.log"
Jul 16 05:41:18.778281 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:18.778237    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5rnd9_5a3625d5-1418-40d6-b16b-1cb8a069163a/konnectivity-agent/0.log"
Jul 16 05:41:18.854800 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:18.854678    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-199.ec2.internal_4fb9095ab1b7993f0bf6be0862d6ba52/haproxy/0.log"
Jul 16 05:41:24.094388 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:24.094310    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw_b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0/extract/0.log"
Jul 16 05:41:24.120957 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:24.120927    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw_b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0/util/0.log"
Jul 16 05:41:24.146201 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:24.146151    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqlhw_b1d4f5d2-7194-41fe-ae56-a0c65f5e5dc0/pull/0.log"
Jul 16 05:41:24.176878 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:24.176810    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv_82315cbd-b165-4552-8f50-301a9738f375/extract/0.log"
Jul 16 05:41:24.200398 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:24.200310    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv_82315cbd-b165-4552-8f50-301a9738f375/util/0.log"
Jul 16 05:41:24.228526 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:24.228492    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j6gv_82315cbd-b165-4552-8f50-301a9738f375/pull/0.log"
Jul 16 05:41:24.260860 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:24.260831    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7_67b252d5-f05a-405e-8f7c-42c087adc22d/extract/0.log"
Jul 16 05:41:24.287302 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:24.287267    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7_67b252d5-f05a-405e-8f7c-42c087adc22d/util/0.log"
Jul 16 05:41:24.311327 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:24.311298    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkdb7_67b252d5-f05a-405e-8f7c-42c087adc22d/pull/0.log"
Jul 16 05:41:24.347970 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:24.347886    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g_560c797e-c2b3-41f5-917a-9dcfc341e9cb/extract/0.log"
Jul 16 05:41:24.375827 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:24.375796    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g_560c797e-c2b3-41f5-917a-9dcfc341e9cb/util/0.log"
Jul 16 05:41:24.402153 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:24.402117    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vmf2g_560c797e-c2b3-41f5-917a-9dcfc341e9cb/pull/0.log"
Jul 16 05:41:24.852762 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:24.852721    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7fc69f979d-ltt5g_c6c86211-b48d-46da-8ad0-3a0a28759bb9/authorino/0.log"
Jul 16 05:41:25.247024 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:25.246985    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-78gs7_65fca02e-3e0b-4e0d-a5a3-60ca94b6f1cb/limitador/0.log"
Jul 16 05:41:27.128036 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:27.127997    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-k6lpx_ba263a6a-62d7-412e-9459-04f53586cdbb/cluster-monitoring-operator/0.log"
Jul 16 05:41:27.163810 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:27.163749    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-9zsxd_ea86cd3f-b598-443a-8d70-cb2fc310aecd/kube-state-metrics/0.log"
Jul 16 05:41:27.186706 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:27.186637    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-9zsxd_ea86cd3f-b598-443a-8d70-cb2fc310aecd/kube-rbac-proxy-main/0.log"
Jul 16 05:41:27.215808 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:27.215773    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-9zsxd_ea86cd3f-b598-443a-8d70-cb2fc310aecd/kube-rbac-proxy-self/0.log"
Jul 16 05:41:27.247837 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:27.247771    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b6b48b445-m7brf_c830977f-610a-4d9e-bf1e-a2d93fe4a54f/metrics-server/0.log"
Jul 16 05:41:27.278536 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:27.278508    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-sdlvv_7412f889-1312-4254-ba2d-fbc59c89b468/monitoring-plugin/0.log"
Jul 16 05:41:27.311883 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:27.311842    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7rtjs_1ce6ade3-c5ef-48dc-aec9-85277e8472fd/node-exporter/0.log"
Jul 16 05:41:27.334730 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:27.334695    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7rtjs_1ce6ade3-c5ef-48dc-aec9-85277e8472fd/kube-rbac-proxy/0.log"
Jul 16 05:41:27.358404 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:27.358368    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7rtjs_1ce6ade3-c5ef-48dc-aec9-85277e8472fd/init-textfile/0.log"
Jul 16 05:41:27.855333 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:27.855279    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-w6c87_84c30e52-23b6-4763-811a-1c81d8712e01/prometheus-operator-admission-webhook/0.log"
Jul 16 05:41:27.889907 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:27.889872    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-df4686d48-qdq7s_4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12/telemeter-client/0.log"
Jul 16 05:41:27.911836 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:27.911800    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-df4686d48-qdq7s_4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12/reload/0.log"
Jul 16 05:41:27.936478 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:27.936436    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-df4686d48-qdq7s_4a2d7b1c-ed2e-49d4-a0e2-9cb6d6187a12/kube-rbac-proxy/0.log"
Jul 16 05:41:27.970815 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:27.970777    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c68d77877-fkzdg_4a6fa390-9d6a-4590-97a0-baa59f13b954/thanos-query/0.log"
Jul 16 05:41:28.000543 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:28.000462    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c68d77877-fkzdg_4a6fa390-9d6a-4590-97a0-baa59f13b954/kube-rbac-proxy-web/0.log"
Jul 16 05:41:28.024535 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:28.024482    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c68d77877-fkzdg_4a6fa390-9d6a-4590-97a0-baa59f13b954/kube-rbac-proxy/0.log"
Jul 16 05:41:28.051188 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:28.051155    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c68d77877-fkzdg_4a6fa390-9d6a-4590-97a0-baa59f13b954/prom-label-proxy/0.log"
Jul 16 05:41:28.075515 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:28.075486    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c68d77877-fkzdg_4a6fa390-9d6a-4590-97a0-baa59f13b954/kube-rbac-proxy-rules/0.log"
Jul 16 05:41:28.099469 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:28.099438    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c68d77877-fkzdg_4a6fa390-9d6a-4590-97a0-baa59f13b954/kube-rbac-proxy-metrics/0.log"
Jul 16 05:41:29.293140 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.293109    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-ldqtw_5dedf531-2032-458b-8003-a88d1379a808/networking-console-plugin/0.log"
Jul 16 05:41:29.328757 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.328719    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"]
Jul 16 05:41:29.338310 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.338274    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.339121 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.339091    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"]
Jul 16 05:41:29.416617 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.415573    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/251f708d-62ce-4844-a1b4-3368304edef9-sys\") pod \"perf-node-gather-daemonset-wvt2h\" (UID: \"251f708d-62ce-4844-a1b4-3368304edef9\") " pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.416617 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.415675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/251f708d-62ce-4844-a1b4-3368304edef9-proc\") pod \"perf-node-gather-daemonset-wvt2h\" (UID: \"251f708d-62ce-4844-a1b4-3368304edef9\") " pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.416617 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.415706    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gtpl\" (UniqueName: \"kubernetes.io/projected/251f708d-62ce-4844-a1b4-3368304edef9-kube-api-access-9gtpl\") pod \"perf-node-gather-daemonset-wvt2h\" (UID: \"251f708d-62ce-4844-a1b4-3368304edef9\") " pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.416617 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.415776    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/251f708d-62ce-4844-a1b4-3368304edef9-podres\") pod \"perf-node-gather-daemonset-wvt2h\" (UID: \"251f708d-62ce-4844-a1b4-3368304edef9\") " pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.416617 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.415847    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/251f708d-62ce-4844-a1b4-3368304edef9-lib-modules\") pod \"perf-node-gather-daemonset-wvt2h\" (UID: \"251f708d-62ce-4844-a1b4-3368304edef9\") " pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.516569 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.516508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/251f708d-62ce-4844-a1b4-3368304edef9-sys\") pod \"perf-node-gather-daemonset-wvt2h\" (UID: \"251f708d-62ce-4844-a1b4-3368304edef9\") " pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.516805 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.516583    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/251f708d-62ce-4844-a1b4-3368304edef9-proc\") pod \"perf-node-gather-daemonset-wvt2h\" (UID: \"251f708d-62ce-4844-a1b4-3368304edef9\") " pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.516805 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.516638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gtpl\" (UniqueName: \"kubernetes.io/projected/251f708d-62ce-4844-a1b4-3368304edef9-kube-api-access-9gtpl\") pod \"perf-node-gather-daemonset-wvt2h\" (UID: \"251f708d-62ce-4844-a1b4-3368304edef9\") " pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.516805 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.516701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/251f708d-62ce-4844-a1b4-3368304edef9-podres\") pod \"perf-node-gather-daemonset-wvt2h\" (UID: \"251f708d-62ce-4844-a1b4-3368304edef9\") " pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.516805 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.516720    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/251f708d-62ce-4844-a1b4-3368304edef9-proc\") pod \"perf-node-gather-daemonset-wvt2h\" (UID: \"251f708d-62ce-4844-a1b4-3368304edef9\") " pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.516805 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.516791    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/251f708d-62ce-4844-a1b4-3368304edef9-lib-modules\") pod \"perf-node-gather-daemonset-wvt2h\" (UID: \"251f708d-62ce-4844-a1b4-3368304edef9\") " pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.517076 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.516970    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/251f708d-62ce-4844-a1b4-3368304edef9-lib-modules\") pod \"perf-node-gather-daemonset-wvt2h\" (UID: \"251f708d-62ce-4844-a1b4-3368304edef9\") " pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.517076 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.516973    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/251f708d-62ce-4844-a1b4-3368304edef9-podres\") pod \"perf-node-gather-daemonset-wvt2h\" (UID: \"251f708d-62ce-4844-a1b4-3368304edef9\") " pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.517192 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.516700    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/251f708d-62ce-4844-a1b4-3368304edef9-sys\") pod \"perf-node-gather-daemonset-wvt2h\" (UID: \"251f708d-62ce-4844-a1b4-3368304edef9\") " pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.525166 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.525126    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gtpl\" (UniqueName: \"kubernetes.io/projected/251f708d-62ce-4844-a1b4-3368304edef9-kube-api-access-9gtpl\") pod \"perf-node-gather-daemonset-wvt2h\" (UID: \"251f708d-62ce-4844-a1b4-3368304edef9\") " pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.669385 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.667042    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:29.909444 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:29.908750    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"]
Jul 16 05:41:29.911615 ip-10-0-138-199 kubenswrapper[2565]: W0716 05:41:29.911502    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod251f708d_62ce_4844_a1b4_3368304edef9.slice/crio-03ad6490e877217f2b304b9b76709e7e727fd858c7dfe110e55ab999dd09e724 WatchSource:0}: Error finding container 03ad6490e877217f2b304b9b76709e7e727fd858c7dfe110e55ab999dd09e724: Status 404 returned error can't find the container with id 03ad6490e877217f2b304b9b76709e7e727fd858c7dfe110e55ab999dd09e724
Jul 16 05:41:30.071861 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:30.071824    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/1.log"
Jul 16 05:41:30.079953 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:30.079574    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-msc9n_9582f0df-040b-4ff8-8eca-53207122af5b/console-operator/2.log"
Jul 16 05:41:30.278505 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:30.278457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h" event={"ID":"251f708d-62ce-4844-a1b4-3368304edef9","Type":"ContainerStarted","Data":"763fcaf29a08132a1b5ff0c4dd145483c6aaa9f044ed634f675df027ac89a96a"}
Jul 16 05:41:30.278717 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:30.278610    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h" event={"ID":"251f708d-62ce-4844-a1b4-3368304edef9","Type":"ContainerStarted","Data":"03ad6490e877217f2b304b9b76709e7e727fd858c7dfe110e55ab999dd09e724"}
Jul 16 05:41:30.278934 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:30.278905    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:30.296253 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:30.296168    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h" podStartSLOduration=1.296139681 podStartE2EDuration="1.296139681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 05:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:41:30.293207989 +0000 UTC m=+3093.709163064" watchObservedRunningTime="2026-07-16 05:41:30.296139681 +0000 UTC m=+3093.712094760"
Jul 16 05:41:30.651396 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:30.651281    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-wbzhk_00dab318-0248-4bf4-b18b-d439c871101d/download-server/0.log"
Jul 16 05:41:32.403174 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:32.403147    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tfrst_d2e8927f-33f5-4cce-a369-b7736e189b57/dns/0.log"
Jul 16 05:41:32.426722 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:32.426694    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tfrst_d2e8927f-33f5-4cce-a369-b7736e189b57/kube-rbac-proxy/0.log"
Jul 16 05:41:32.475764 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:32.475733    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cbrhk_4d3ed278-258b-4590-846b-c11283cbbba3/dns-node-resolver/0.log"
Jul 16 05:41:33.144401 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:33.144370    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sslfh_a2066081-de65-45f1-a46c-0f456249f49e/node-ca/0.log"
Jul 16 05:41:34.281774 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:34.281723    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c7fc4f54f-mpm4d_a972592d-555b-436e-bee8-2ae1a60f45e0/kube-auth-proxy/0.log"
Jul 16 05:41:34.415403 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:34.415364    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-pn9n8_f3325934-58e9-43c1-a9c3-529eca6de6fd/payload-processing/0.log"
Jul 16 05:41:35.165646 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:35.165611    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dk6m5_31176ca2-45ab-4eb0-abc7-b4ceae534396/serve-healthcheck-canary/0.log"
Jul 16 05:41:35.710321 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:35.710275    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8pp6l_7a68f257-3410-4b65-b926-58ff206520a5/kube-rbac-proxy/0.log"
Jul 16 05:41:35.732708 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:35.732666    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8pp6l_7a68f257-3410-4b65-b926-58ff206520a5/exporter/0.log"
Jul 16 05:41:35.756252 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:35.756220    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8pp6l_7a68f257-3410-4b65-b926-58ff206520a5/extractor/0.log"
Jul 16 05:41:36.303140 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:36.303095    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4w7ht/perf-node-gather-daemonset-wvt2h"
Jul 16 05:41:39.176509 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:39.176474    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dddb677c8-xhhkb_df4937bc-c6ba-494d-83a0-b261dc5ef290/manager/0.log"
Jul 16 05:41:39.198354 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:39.198327    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dddb677c8-xhhkb_df4937bc-c6ba-494d-83a0-b261dc5ef290/copy-manifests/0.log"
Jul 16 05:41:40.554391 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:40.554310    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8647fcdd7d-b9cw4_53181c67-a04a-4151-96eb-5ca9bcdcec30/manager/0.log"
Jul 16 05:41:40.581881 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:40.581841    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-4ntkw_9c3f6ce1-85a8-4bae-94fd-0052112bc5be/openshift-lws-operator/0.log"
Jul 16 05:41:41.156654 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:41.156625    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c7fc4f54f-mpm4d_a972592d-555b-436e-bee8-2ae1a60f45e0/kube-auth-proxy/0.log"
Jul 16 05:41:41.265506 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:41.265475    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-pn9n8_f3325934-58e9-43c1-a9c3-529eca6de6fd/payload-processing/0.log"
Jul 16 05:41:42.571420 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:42.571381    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c7fc4f54f-mpm4d_a972592d-555b-436e-bee8-2ae1a60f45e0/kube-auth-proxy/0.log"
Jul 16 05:41:42.690358 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:42.690304    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-pn9n8_f3325934-58e9-43c1-a9c3-529eca6de6fd/payload-processing/0.log"
Jul 16 05:41:45.972841 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:45.972810    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-2xgh8_95bf5f5b-d8bb-4638-895a-015e35aea09a/kube-storage-version-migrator-operator/1.log"
Jul 16 05:41:45.973928 ip-10-0-138-199 kubenswrapper[2565]: I0716 05:41:45.973906    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-2xgh8_95bf5f5b-d8bb-4638-895a-015e35aea09a/kube-storage-version-migrator-operator/0.log"